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Introduction 

The  concept  of  reduction  in  logistical  footprint  can  logically  be  applied  not  only  to  fighting 
forces  but  also  to  mobile  surgical  facilities  that  might  be  co-deployed.  Reduction  in  the  number 
of  personnel  required  to  staff  mobile  surgical  facilities  would  be  a  way  of  achieving  this 
logistical  reduction.  At  the  beginning  of  this  contract,  we  had  recently  completed  preliminary 
work  in  constructing  a  prototype  robotic  surgical  device  that  can  perform,  on  a  very  limited 
basis,  the  essential  job  functions  of  the  surgical  scrub  technician  in  the  operating  room.  In 
essence,  the  robot  is  a  computer  control  system  connected  to  a  digital  camera  and  a  mechanical 
arm  with  a  gripper.  The  computer  control  system  also  has  a  voice  recognition  system  to  listen  to 
verbal  requests  for  an  instrument  from  the  surgeon.  The  robot  uses  machine-vision  via  the 
camera  to  locate  and  identify  surgical  instruments  and  also  to  compute  their  orientation.  When 
the  surgeon  requests  an  instrument,  the  mechanical  arm  delivers  that  instrument  to  the  surgeon. 
WTien  the  surgeon  is  finished  with  the  instrument,  the  arm  is  sent  to  the  coordinates  of  the 
instrument  lying  out  on  the  surgical  field,  to  retrieve  it  and  return  it  to  the  robot’s  Mayo  stand. 
The  work  done  under  this  contract  was  to  improve  two  core  capabilities  of  the  machine  so  that  it 
comes  closer  to  being  able  to  replace  a  human  scrub  technician  for  the  performance  of  a  basic 
repertoire  of  surgical  procedures.  The  two  specific  goals  in  the  statement  of  work  were  1)  to 
improve  our  robot’s  vision  capabilities  so  that  it  can  correctly  recognize  twelve  instruments  with 
98%  accuracy  and  2)  to  make  the  robot  98%  reliable  in  physically  retrieving  the  instruments  that 
it  has  recognized.  Previously,  the  vision  routines  were  able  to  detect  and  distinguish  four 
instruments  with  an  80%  accuracy  rate  and  the  retrieval  accuracy  rate  was  rate  85%.  These 
systems  are  at  the  core  of  the  robot’s  basic  function,  and  errors  must  be  reasonably  infrequent. 
We  were  entirely  successful  in  meeting  the  first  goal,  and  came  close  to  meeting  the  second  goal. 
Combined  with  additional  work  that  was  done  concurrently,  the  contract  work  has  the  important 
result  that  the  robot’s  development  has  progressed  such  that  we  expect  to  meet  our  ultimate  goal 
of  clinical  deployment  in  the  coming  year  of  2005. 

Body 

The  body  of  this  report  is  divided  into  two  main  sections.  The  first  section  is  a  complete 
description  of  the  research  accomplishments  to  date  with  respect  to  the  approved  Statement 
of  Work  for  DAMD17-03-C-0083.  The  specific  tasks  and  results  are  described  in  the  sub¬ 
sections  following  this  one,  entitled  Vision  System  and  Motion  System.  The  second  section  is  a 
Narrative  Description  of  the  Development  of  the  Penelope  System.  This  narrative  description 
shows  the  full  scope  of  work  that  occurred  during  this  time  period,  including  the  parallel  threads 
of  development  and  accomplishment  during  this  time.  This  narrative  is  provided  in  order  to  give 
context  to  the  specific  tasks  that  were  accomplished  in  fulfillment  of  the  Statement  of  Work.  The 
narrative  section  shows  why  the  specific  work  done  with  TATRC  support  is  important  to  the 
development  of  a  successful  clinical  system. 

Vision  System 

Instrument  Choice 

In  approaching  the  goal  of  recognizing  twelve  instrument  types,  it  was  first  necessary  to  choose 
the  instruments  to  include  in  this  set.  If  possible  it  was  desirable  to  choose  a  set  with  which  one 
could  perform  a  simple  general  surgery  operation  that  Penelope  is  intended  for  use  with.  We  had 
already  compiled  a  database  of  instrument  requests  in  some  basic  operations,  for  use  in 
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evaluating  the  software  that  allows  Penelope  to  predict  the  next  request.  From  this  database  we 
extracted  the  twelve  most  commonly  requested  instruments  in  excisions  of  lipomas  and  cysts: 


Suture  Scissors 
Tooth  Forceps 
Needle  Holder 
Hopkins  Clamp 
Debakey  Forceps 
Brown-Adson  Forceps 
Richardson  Retractor 
Metzenbaum  Scissors 
Allis  Clamp 
Loop  Retractor 
Adson  Clamp 


The  scalpel  is  not  included  in  this  list  although  it  was  in  fact  the  most  commonly  requested 
instrument  of  all,  because  for  safety  reasons  Penelope  will  never  hand  an  exposed  blade  directly 
to  the  surgeon.  If  the  robot  delivers  a  scalpel,  it  will  do  so  in  a  protective  package,  and  since  we 
will  design  such  a  package,  we  can  produce  it  in  any  shape  or  color  so  as  to  be  easily  recognized. 

Vision  Algorithms 

This  section  will  describe  all  of  the  methods  currently  in  use  to  detect  and  identify  instruments  in 
the  robot’s  field  of  view.  After  the  camera  captures  an  image,  the  system  software  takes  it 
through  several  stages  of  analysis,  with  the  goal  of  separating  objects  from  the  background,  then 
measuring  and  identifying  the  objects.  The  progression  through  the  stages  of  analysis  is  shown 
in  Fig.  1,  with  detailed  descriptions  of  each  stage  following. 


t 


Raw  Image 


Screening  and  Blob 
Detection:  White  pixels 
represent  those  determined 
to  be  part  of  the  object. 


Fiducials:  A  dark  line  shows 
the  object’s  line  of  maximal 
symmetry  and  dots  show 
points  from  which  length  and 
width  are  measured. 


Fig.  I.  Screen  captures  of  the  most  important  stages  of  the  vision  processing  for  one  image. 


Raw  Image  Screening 

When  the  system  is  first  turned  on,  an  image  is  captured  of  the  surface  of  the  transfer  zone;  this 
serves  as  the  background  against  which  new  images  are  compared.  As  the  first  step  in 
processing,  the  captured  image  is  compared  to  the  stored  background  image.  Rather  than 
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comparing  individual  pixels  in  each  image,  which  would  be  overly  sensitive,  the  algorithm 
compares  each  pixel  in  the  captured  image  to  a  histogram  that  describes  what  colors  occurred  in 
the  background  image.  Therefore,  to  be  considered  part  of  the  foreground,  a  pixel  in  the 
captured  image  must  be  of  a  color  not  found  anywhere  in  the  background  image.  The  results  of 
this  comparison  are  stored  in  a  binary  image  array,  with  white  pixels  representing  the 
foreground,  and  black  pixels  the  background. 

Blob  Detection 

Once  we  know  which  individual  pixels  may  be  part  of  an  instrument,  we  must  group  adjacent 
and  nearby  ones  together  into  what  we  call  ‘blobs.’  When  a  set  of  pixels  is  part  of  one  blob,  it 
simply  means  they  are  close  enough  together  that  it  is  likely  they  are  all  part  of  the  same  larger 
shape.  The  algorithms  turn  the  screened  binary  image  array  into  an  image  with  a  different  color 
for  each  set  of  pixels  that  represents  a  distinct  blob.  The  largest  of  these  is  taken  to  be  the 
instrument.  Currently  we  assume  that  only  one  instrument  is  on  the  transfer  zone  at  a  time.  At 
present,  the  software  can  deal  with  more  than  one  instrument  but  only  if  they  are  not 
overlapping. 

Fiducials 

Having  the  blob  that  most  likely  represents  the  instrument,  the  remaining  task  is  to  measure  it 
and  determine  the  instrument  type.  The  fiducials  are  measurements  taken  from  the  blob 
produced  by  the  previous  step.  The  system  has  been  trained  to  know  the  measurements  of  each 
instrument,  and  when  a  new  image  is  captured,  it  compares  its  measurements  to  those  of  all  the 
instruments  and  chooses  the  closest  match.  The  fiducials  are  the  crux  of  the  high-level  vision 
system,  and  the  part  which  we  have  spent  the  most  time  refining  to  enable  identification  of  12 
instrument  types.  Currently  five  measurements  are  taken  for  all  instrument  types,  and  two 
specialized  ones  are  only  taken  to  help  distinguish  two  very  similar  instruments.  The  primary 
five  measurements  are  length,  width,  and  the  x-axis,  y-axis,  and  z-axis  moments  of  inertia.  The 
two  specialized  ones  are  ‘tip  width’  and  ‘finger  loop  length.’  These  will  be  discussed  further 
below. 

Acquisition  of  fiducials  is  a  bootstrapping  process  that  uses  each 
piece  of  information  to  obtain  the  next  one.  It  starts  with  the 
center  of  mass,  or  centroid,  which  is  simply  the  pixel  in  the 
center  of  the  blob.  Next  we  divide  the  blob  into  four  quadrants, 
with  the  centroid  as  the  origin.  We  compute  the  centroid  of  the 
pixels  in  each  of  those  quadrants,  and  whichever  is  furthest  from 
the  centroid  becomes  the  secondary  centroid.  We  found  that  by 
this  method,  the  line  connecting  the  centroid  and  the  secondary 
centroid  is  the  instrument’s  line  of  maximal  symmetry.  This  line 
is  the  key  to  the  rest  of  the  fiducials,  because  it  provides  axes 
along  which  to  measure  length  and  width,  as  well  as  the 
moments  of  inertia  (in  fact  the  y-axis  for  the  moment  of  inertia  is 
the  line  of  maximal  symmetry  itself).  Fig.  2  shows  a  detail  of 
the  fiducials  for  the  suture  scissors:  the  two  X’s  are  the  centroid 
and  secondary  centroid,  and  the  dark  line  is  the  line  of  maximal 
symmetry. 


Fig.  2.  Screen  capture 
of  fiducials  markers 
placed  on  an  instrument 
by  the  software. 
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Moments  of  Inertia 


Fig.  3.  The  axes  used  for  moment  of  inertia  measurements 

The  moments  of  inertia  are  measures  of  how  the  weight  of  an  object  is  distributed  around  its 
area.  As  physical  properties,  they  describe  how  an  object  responds  to  rotational  forces  around  its 
axes.  The  moments  are  computed  as  the  average  distance  of  the  pixels  in  the  object  from  each 
axis.  For  our  purposes,  they  give  indications  about  the  object’s  shape  that  are  subtler  than  the 
length  and  width  alone.  Shown  in  Fig.  3  are  illustrations  of  the  three  rotational  axes  as  we  use 
them. 


Tip  Width  and  Finger  Loop  Length 

After  implementing  the  moments  of  inertia  and  performing  tests  to  determine  accuracy  rates;  we 
found  we  could  still  not  distinguish  between  two  particular  instruments,  the  Adson  clamp  and  the 
needle  holder.  Their  profiles  are  simply  too  similar.  To  remedy  this  we  looked  carefully  at  their 
shapes,  and  thought  about  what  could  be  measured  once  it  is  known  that  one  of  these  two 
instruments  is  being  observed.  We  came  up  with  tip  width,  and  finger  loop  length.  The  tip 
width  is  a  measure  of  how  narrow  and  pointed  the  tip  of  the  instrument  is;  our  own  observations, 
and  subsequent  testing,  showed  that  the  Adson  clamp  has  a  sharper  tip  than  the  needle  holder. 
Finger  loop  length  is  a  measure  of  the  size,  along  the  clamp’s  length,  of  the  loops  in  which  the 
surgeon’s  fingers  grip  the  clamp.  It  was  observed  and  subsequently  confirmed  that  the  needle 
holder’s  loops  are  larger  to  a  visually  significant  degree  than  those  of  the  Adson  clamp  (Fig.  4). 


Fig.  4.  In  these  images  the  needle  holder  is  on  the  left,  and  the  Adson  clamp  is  on  the  right.  At 
the  left  and  middle  are  raw  and  screened  images  taken  by  the  system’s  camera.  At  right  are 
details  of  the  tips  and  finger  loops  of  slightly  different  size. 
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Instrument  Identification  Decisions 


The  reason  multiple  criteria  are  necessary,  and  that  similar  but  not  identical  instruments  pose 
problems  for  identification,  is  that  the  measurements  are  never  quite  the  same  over  multiple 
trials.  Unpredictable  shadows  and  reflections,  as  well  as  the  limited  resolution  of  the  camera, 
produce  a  range  of  measurements  over  a  training  set  of  images,  that  the  instrument’s  ‘actual’ 
dimensions  can  be  assumed  to  fall  within.  Similar  instruments  end  up  with  overlapping  ranges, 
and  whenever  a  captured  image’s  measurements  fall  within  the  overlap  area,  the  system  must 
decide  between  the  two  instruments.  This  is  done  by  taking  the  Euclidean  distance  from  the 
current  measurements  to  the  averages  for  each  possible  instrument,  and  choosing  the  shorter 
distance.  In  practice  this  distance  is  in  an  n-dimensional  space  where  n  is  the  number  of  criteria, 
up  to  7  if  all  are  being  used.  For  an  illustrative  example  in  Fig.  5,  we  will  use  only  length  and 
width. 


width 

Fig.  5.  A  possible  decision  scenario.  The  two  rectangles  represent  the  measurement  ranges  for  these 
instruments  in  the  training  set.  Since  the  current  measurement  falls  within  the  overlap  area  but  is  much 
closer  to  the  average  for  the  Adson  clamp,  that  instrument  would  be  chosen  as  the  identification.  Note 
that  the  average  is  not  necessarily  directly  in  the  middle  of  the  minima  and  maxima,  although  it  is  often 
close. 


Confidence  Interval 

After  the  first  rounds  of  testing,  it  was  found  that  no  matter  how  much  each  instrument  was 
trained  beforehand,  some  of  the  measurements  taken  in  testing  would  always  be  slightly  outside 
the  observed  range.  In  this  situation  the  system  would  come  up  with  no  identification,  even  if 
the  measurement  was  still  quite  close  to  one  Instrument  and  nowhere  near  any  others.  To  combat 
this  we  implemented  a  statistics  concept  called  the  confidence  interval,  which  is  essentially  a 
way  of  expanding  the  ranges  from  what  has  been  observed  so  far,  to  try  to  include  what  will  be 
observed  in  the  future.  Although  such  an  expansion  enlarges  the  overlap  area  for  similar 
instruments,  the  distance-to-averages  method  for  decisions  prevents  this  from  damaging 
accuracy  rates.  The  confidence  interval  is  calculated  thus: 


i  =  kx 

v  n 


where  it  is  a  precision  value  indicating  the  percentage  of  future  measurements  that  should  fall 
within  the  interval,  a  is  the  standard  deviation  of  the  training  set,  and  n  is  the  sample  size  of  the 
training  set. 
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Vision  System  Testing 

In  order  to  test  our  instrument  identification  algorithms  and  determine  how  best  to  improve  them, 
we  designed  the  following  procedure.  First,  to  isolate  the  algorithms  as  a  variable,  we  built  a 
light-tight  box  around  the  system’s  camera  and  installed  a  light  inside  it.  We  had  found  that  with 
the  wide  window  in  our  lab,  the  powerful  sunlight  changed  the  images  so  much  that  a  test  done 
during  the  day  and  taking  several  hours  could  not  be  reliable.  Since  operating  rooms  do  not  have 
windows  for  similar  reasons,  we  assume  this  to  be  a  reasonable  level  of  control. 

As  a  training  set,  each  instrument  is  placed  in  a  random  position  under  the  camera  and  the  system 
is  told  that  what  it  sees  is  that  instrument,  taking  the  various  measurements  used  for 
identification.  Each  instrument  is  trained  approximately  25  times,  depending  on  the  observed 
variation  in  the  measurements.  Since  we  did  not  change  the  way  measurements  were  taken,  we 
were  able  to  use  a  cumulative  training  set  for  all  of  our  tests. 

In  the  testing  phase,  each  instrument  is  placed  under  the  camera  20  times  and  an  attempt  is  made 
at  identification.  Whenever  the  system  fails  to  recognize  that  a  given  instrument  is  there,  that 
instrument’s  “true-negative”  error  count  is  incremented.  If  the  system  thinks  an  instrument  is 
there  which  is  actually  not,  that  instrument’s  “false-positive”  error  count  is  added  to.  A  case  of 
mistaking  one  instrument  for  another  counts  as  one  error  of  each  type.  As  an  example,  if  the 
Metzenbaum  scissors  were  placed  under  the  camera,  and  the  system  reported  recognition  of  the 
suture  scissors  instead,  it  would  be  a  true-negative  error  for  the  Metzenbaum  scissors,  and  a 
false-positive  error  for  the  suture  scissors.  At  the  end  of  the  test,  the  accuracy  rate  is  computed 
as  the  number  of  true-negative  errors  over  the  number  of  trials.  We  do  not  include  false-positive 
errors  in  the  accuracy  rate,  since  this  would  be  in  effect  counting  some  errors  twice. 

In  the  first  round  of  testing,  we  achieved  a  recognition  rate  of  81.6%.  The  algorithms  at  this 
point  measured  length,  width,  and  y-axis  moment  of  inertia,  and  simply  checked  whether  or  not  a 
measurement  was  in  the  range  of  values  in  each  instrument’s  training  set.  We  had  determined 
that  the  other  two  moments  of  inertia,  white  reliable  measurements,  did  not  offer  any  additional 
level  of  distinction  between  similar  instruments.  Most  of  the  errors  in  this  test  occurred  when  a 
measurement  was  just  slightly  outside  the  training  set’s  range,  causing  the  instrument  to  fall  out 
of  consideration.  At  this  point  we  implemented  the  confidence  interval.  After  this  improvement, 
in  the  second  round  of  testing  we  achieved  an  accuracy  rate  of  91.6%.  The  next  major 
enhancement  was  to  fix  a  problem  that  occurred  when  instruments  were  angled  horizontally  or 
vertically,  aligned  with  the  camera’s  coordinate  system.  This  accounted  for  approximately  1/3  of 
the  errors  observed  in  the  second  test.  To  fix  it,  we  wrote  an  algorithm  to  detect  when  an 
instrument  was  at  such  an  angle,  and  with  that  knowledge  use  a  slightly  different  method  to  take 
the  measurements.  The  third  test  incorporated  both  this  improvement  and  the  ability  to 
recognize  certain  instruments,  such  as  forceps  and  retractors,  in  multiple  profiles  based  on  how 
they  are  laid  down.  No  angle  errors  occurred,  and  the  accuracy  rate  rose  to  93.75%. 

Two  sources  of  error  remained  at  this  point,  each  contributing  about  half  of  the  15  errors.  Errors 
having  to  do  with  out-of-range  measurements  still  occurred  more  often  than  expected.  This  was 
resolved  simply  by  increasing  the  precision  value  k  in  the  confidence  interval  equation.  The 
other  source  of  error  was  confusion  between  the  two  most  similar  instruments,  the  Adson  clamp 
and  the  needle  holder.  For  these  we  implemented  tip  width  and  finger  loop  length  as  further 
distinguishing  factors.  After  these  improvements  we  performed  a  fourth  round  of  testing,  in 
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which  we  achieved  our  goal  with  a  98.3%  accuracy  rate. 


Vision  System  Resuits 

Our  basic  system  for  visually  recognizing  surgical  instruments  was  improved  to  achieve  our 
stated  technical  goal  of  98%  accuracy  rate  applied  to  twelve  surgical  instruments.  This  overall 
result  came  from  a  combination  of  several  improvements,  including  initially  adding  additional 
criteria  (moments  of  inertia),  allowing  for  a  certain  amount  of  statistical  uncertainty  in  the 
measurements  (confidence  interval),  correcting  software  deficiencies  that  occurred  at  certain 
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Fig.  6.  Summary  of  the  accuracy  rates  for  each  test  round,  and  the  improvements  made 
between  each. 

orientations,  adding  capability  to  recognize  the  same  instrument  in  different  profiles,  and  then 
adding  certain  specific  criteria  that  were  applied  by  additional  code  when  certain  “look  alike” 
instruments  were  encountered.  These  results  are  graphically  shown  in  Fig.  6. 

Vision  System:  Future  Work 

The  vision  system  will  undergo  several  changes  as  Penelope  gets  closer  to  a  clinical  version; 
some  of  these  will  improve  performance,  others  will  introduce  new  challenges.  The  camera  will 
eventually  be  positioned  higher  up,  able  to  see  the  Mayo  stand  and  staging  zone  as  well  as  the 
transfer  zone.  This  will  make  all  measurements  smaller  and  closer  together,  creating  the  chance 
for  more  confusion  between  instruments.  It  can  be  alleviated  by  using  a  larger  amount  of  the 
camera’s  resolution  than  we  currently  use.  We  will  also  have  to  deal  with  non-instrument 
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objects  entering  the  field  of  view,  such  as  sponges,  syringes  containing  anesthetic,  and  gloved 
hands  of  assistants  or  observers.  On  the  other  hand,  we  will  be  able  to  use  the  knowledge  of 
what  instruments  are  in  use  to  simplify  identification  decisions.  If  there  is  uncertainty  over 
whether  an  instrument  is  an  Adson  clamp  or  a  needle  holder,  but  only  a  needle  holder  is  in  use, 
then  the  uncertainty  is  eliminated.  Of  course  the  biggest  change  will  be  the  size  of  the 
instrument  set,  which  will  eventually  expand  to  42  types  as  found  on  the  Minor  Surgical  Set.  We 
are  confident  that  the  method  we  have  used  so  far,  adding  specific  identification  criteria  where 
necessary,  will  allow  us  to  maintain  acceptable  accuracy  rates  as  we  add  more  instruments  to 
Penelope’s  repertoire. 


Motion  System 

Motion  Algorithms 

The  robot’s  motion  is  controlled  on  the  software  side  by  a  physics-based  simulator,  and  on  the 
hardware  side  by  a  USB-connected  microcontroller  and  the  servo  motors.  When  the  arm  is  told 
to  move  to  a  point,  the  simulator  computes  the  necessary  movement  of  each  motor  to  bring  the 
arm  closer  to  its  destination,  and  sends  a  command  to  the  motors.  These  commands  are 
individual  time  steps  in  the  overall  movement  toward  the  destination,  and  are  sent  to  the  motors 
every  20  milliseconds.  There  are  two  motors  affecting  the  height  of  the  arm,  at  the  elbow  and 
shoulder,  and  another  at  the  shoulder  for  the  horizontal  swivel. 

The  term  “physics  based”  is  used  since  the  simulator’s  modeling  of  the  arm  is  based  on 
principles  of  rigid  body  rotation  under  the  influence  of  applied  forces,  as  calculated  by  applying 
Newton’s  second  law  of  motion:  Force  =  Mass  x  Acceleration.  This  equation  is  numerically 
integrated  to  solve  for  the  acceleration,  velocity  and  position  of  each  rigid  body,  using 
quaternions  to  represent  the  3D  orientation  of  each  segment  of  the  arm.  Since  all  of  the  arm’s 
joints  are  effectively  hinges  with  an  angle  controlled  by  the  motor,  each  segment  is  defined  as  a 
hinged  body,  which  is  a  type  of  rigid  body.  Each  hinged  body  is  defined  by  its  constant  length, 
width  and  height,  mass,  and  center  of  gravity,  as  well  as  the  axis  on  which  it  rotates,  called  the 
hinge-pin.  The  hinged  body  also  has  two  vectors  used  in  calculations,  the  ‘pointer,’  which  runs 
from  the  hinge  point  through  the  center  of  gravity,  and  the  ‘hinge  normal,’  which  is  normal  to 


Fig.  7.  A  hinged  body  before  and  after  a  90  degree  rotation  about  the  hinge-pin.  The  body  is 
defined  by  the  line  segment  ab,  with  center  of  gravity  eg. 
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both  the  hinge  pin  and  the  pointer.  Shown  in  Fig.  7  is  a  hinged  body  rotating  90  degrees  about 
the  hinge -pin  axis. 

In  determining  how  much  each  motor  should  do  for  the  arm  to  execute  the  desired  overall 
movement,  the  simulator  works  its  way  up  from  the  elbow  to  the  shoulder,  trying  to  do  as  much 
as  possible  with  the  smallest  part  of  the  arm.  This  is  analogous  to  how,  when  typing  for 
example,  we  mostly  move  the  fingers  to  reach  the  keys,  rather  than  moving  the  entire  arm  one 
inch  over.  At  each  hinged  body,  the  forces  to  be  put  on  the  arm  as  a  whole  are  filtered  to  only 
include  those  along  the  hinge  normal,  so  that  the  motor  is  not  being  asked  to  move  in  a  way  that 
it  cannot.  The  forces  that  cannot  be  handled  by  one  hinged  body  are  passed  up  to  its  ‘parent,’ 
further  up  the  arm.  When  a  hinged  body  is  moved,  the  quaternion  describing  its  orientation  is 
updated  to  reflect  it.  When  a  hinged  body’s  parent  moves,  its  position  must  be  updated  so  that  it 
is  still  ‘attached’  to  the  rest  of  the  arm.  This  process  of  moving  each  hinged  body  and  updating 
is  performed  at  each  simulation  time  step,  and  over  many  steps  forms  an  overall  arm  movement. 
One  iteration  of  the  process  with  a  simplified  arm  is  shown  in  the  following  sequence. 


A  simplified  arm  before  a  movement 
toward  the  target.  Hj  and  ^{2  are 
the  hinge  points  of  hinge  bodies  rod, 
and  rodj 


H2  rodj 


target 


F,  and  Fj  are  the  component  forces 
of¥  to  be  put  on  Hi  and  for  a 
movement  toward  the  target. 


Fig.  8.  (in  four  parts)  Algorithm  for  motion  of  arm  made  of  two  rigid  bodies  (“rods”),  which 
are  hinged  and  subjected  to  external  forces.  The  algorithm  shows  how  the  hinged  relationship 
is  preserved. 
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The  forces  are  applied  to  both  hinge 
bodies.  The  new  hinge  point  ii\is  a 
result  of  the  slew  force  applied  to  Hi- 


H2  is  moved  to  compensate  for  the 
slew  force,  and  a  step  toward  the 
target  is  completed 


The  calculations  involved  in  each  step  of  the  simulator  can  be  summarized  as  follows: 

1:  Calculate  total  moment  of  forces.  Having  this  value  enables  us  to  evaluate  changes  in  angular 


velocity. 

Mh  =  2(Ri  X  (Fb  •  Nb)Nb) 

Mh 

Legend 

total  moment  of  force  about  hinge 

2:  Update  angular  velocity  of  the  hinged  body  using 

Nb 

point  h 

hinge  normal  vector 

Euler’s  method. 

Fi 

force  applied  to  each  hinged  body 

Ri 

moment  arm  of  application  of 

(0  =  0)+  c/tIh-i(Mh  -  (o)  X  Ih((o))) 

force  Fi 

Fb 

Fi  rotated  from  world  into 

3:  Update  quaternion  of  hinged  body  for  motion  about 

body  coordinates 

hinge  using  Poinsot’s  theorem.  The  ability  to  update 

Ih 

moment  of  inertia  about  h 

quaternions  using  this  elegant  equation  is  the  reason  for 

(0 

angular  velocity  about  h 

using  them  to  represent  orientation,  instead  of  rotational 

0)’ 

slew  angular  velocity 

matrices. 

q 

hinged  body  quaternion 

dt 

dt 

change  in  time  over  one  simulation 

q  =  q 

time-step 

4:  Update  quaternion  for  slew  motion  of  parent.  We  also  update  the  ‘pointer’  vector  that 
indicates  the  hinged  body’s  position,  by  setting  it  to  an  anchor  point  that  is  attached  to  the  parent 
hinged  body. 

=  ‘I 

At  this  point,  we  can  use  the  new  angles  of  the  simulated  hinged  bodies  to  send  commands  to  the 
motors  driving  the  real  physical  robot’s  joints.  In  the  case  of  servos,  we  only  can  use  position 
commands,  but  with  stepper  motors  we  can  also  send  velocity  and  acceleration  commands  to  the 
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motors  as  well. 


Motion  System  Testing 

We  tested  the  current  robot’s  reliability  rate  in  retrieving  an  instrument  from  the  transfer  zone 
and  placing  it  on  the  Mayo  stand.  In  order  to  isolate  the  motion  control  system  as  the  variable  to 
the  greatest  extent  possible,  we  set  several  guidelines.  If  we  could  see  from  the  vision  system’s 
output  that  the  instrument’s  blob  had  not  been  clearly  identified,  and  this  was  the  cause  of  a 
failure,  the  trial  was  thrown  out.  As  long  as  the  instrument  ended  up  on  the  Mayo  stand,  the  trial 
was  counted  as  a  success.  The  trial  was  a  failure  if  the  instrument  was  not  picked  up  at  all,  or  if 
it  was  dropped  prematurely  and  did  not  land  on  the  Mayo  stand.  We  only  used  one  instrument 
for  these  tests,  the  Hopkins  clamp,  and  tested  it  100  times. 

Motion  System  Results 

The  result  was  that  the  robot  successfully  retrieved  the  clamp  94  times,  and  the  other  b  it  did  not 
pick  up  the  instrument  at  all.  Although  this  does  not  meet  our  stated  goal  of  98%,  we  have  for 
several  reasons  decided  not  to  expend  more  effort  toward  improving  the  robot’s  mechanics  at 
this  time.  First,  we  observed  that  all  6  errors  occurred  when  the  instrument  was  in  a  certain  area 
of  the  transfer  zone,  furthest  away  from  the  arm,  and  the  arm’s  error  was  the  same  each  time. 
This  indicates  that  it  is  likely  a  problem  of  calibration  between  the  arm  and  the  camera  or  of  the 
magnet  being  so  far  out  of  the  ideal  perpendicular  orientation  (due  to  fixed  magnet  angle),  and 
not  a  flaw  with  the  motion  control  system.  Second,  the  new  version  of  the  arm  has  been 
constructed  and  is  being  integrated  into  robot’s  operating  system. 

Motion  System:  Future  Work 

The  new  arm  has  been  entirely  redesigned,  has  been  assembled  and  is  now  being  integrated  into 
the  software.  Its  motion  is  produced  by  stepper  motors  which  should  allow  positional  resolution 
meeting  or  exceeding  our  requirements.  Engineering  calculations  indicate  that  angular  resolution 
of  the  joints  of  the  arm  will  be  0.045  degree,  exceeding  our  clinical  requirements.  The  arm  will 
also  incorporate  position  encoders  that  will  allow  the  software  to  detect  discrepancies  between 
commanded  and  achieved  position.  In  addition,  P-3  will  have  a  feedback  mechanism  to  tell 
when  there  is  an  instrument  on  the  gripper.  This  way,  if  an  instrument  is  missed  Penelope  will 
be  aware  of  it  before  it  causes  any  further  errors.  A  key  feature  of  the  new  arm,  compared  to  the 
version  of  the  arm  on  P-2.5,  is  that  this  arm  has  a  “wrist”.  On  the  previous  arms,  in  an  effort  to 
save  weight,  the  wrist  degree-of -freedom  was  not  present  although  these  arms  did  have  the 
ability  to  rotate  instruments.  The  angle  of  the  magnet  relative  to  the  axis  of  the  forearm  was 
chosen  to  be  the  best  compromise  to  provide  (approximately)  perpendicular  orientation  of  the 
magnet  to  the  instruments  over  the  greatest  area  on  the  transfer  zone.  However,  at  the  corners  of 
the  transfer  zone,  the  magnet  would  be  coming  at  a  suboptimal  angle  and  this  we  found  was  a 
contributing  reason  for  the  less  than  desired  overall  retrieval  rate  of  94%.  In  the  new  arm,  the 
wrist  produces  both  rotational  and  flexion-extension  motion,  but  weighs  about  the  same  as  the 
previous  wrist  since  the  motion  is  transmitted  to  a  differential  gear  arrangement  via  drive  belts 
running  entirely  within  the  carbon  fiber  tubes  of  the  forearm  and  upper  arm.  The  two  stepper 
motors  producing  this  wrist  motion  are  mounted  proximal  near  the  shoulder  joint  to  avoid  the 
problems  of  weight  carried  distally  on  the  arm.  The  wrist  allows  the  magnet  to  be  oriented 
perfectly  perpendicularly  to  the  instruments  regardless  of  their  location  on  the  transfer  zone  or 
Mayo  stand. 
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Fig,  9,  New  arm  for  Penelope  3  which  uses  more 
precise  stepper  motor  actuators  than  previous  arm, 
and  also  has  an  additional  ''wrist''  degree-of-freedom 
so  that  the  magnet  will  be  able  to  always  approach 
instruments  perpendicularly. 


Fig,  10.  Differential  gear  system 
produces  combined  rotation  and 
flexion-extension  of  the  wrist. 
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Narrative  of  the  Development  of  the  Penelope  System 


Penelope  1  and  Penelope  2 

At  the  time  of  writing  in  August  of  2002  of 
our  proposal  entitled  “Robotic  Replacement 
for  the  Scrub  Technician”,  we  had  finished 
Penelope  1  (“P-1”)  and  were  building 
Penelope  2  (“P-2”).  As  it  was  pictured  in 
the  original  proposal,  P-2  was  incomplete, 
lacking  a  functional  gripper. 

P-2  was  completed  in  time  for  a 
demonstration  video,  in  November  of  2002, 
which  was  seen  by  the  TATRC  review 
committee.  Compared  to  P-1,  P-2  had 
many  physical  improvements  but  the 
software  was  still  somewhat  slow  and 
required  a  large  dual -processor  computer  to 
run  even  tolerably  well.  Also,  the  camera 
system  was  not  well  integrated  into  the  main 
body  of  the  software,  causing  occasional 
system  crashes. 


The  physical  improvements  for  P-2 
arose  out  of  more  precise 
construction  and  from  servomotors 
which  were  larger  and  more 
accurate.  These  improvements 
promised  to  improve  its  physical 
accuracy  for  retrieving  instruments. 
Additional  work  on  P-2  was  done  in 
order  to  make  the  robot  more 
presentable  for  the  TATRC  exhibit 
at  the  American  Telemedicine 
Association  Meeting  (April,  2003). 
This  work  consisted  of  a  major 
software  overhaul  and  mechanical 
upgrades  to  the  arm  as  well  as  a 
physical  repackaging  of  the  robot, 
as  explained  in  the  next  sections. 
This  work  resulted  in  the  Penelope 
2.5,  which  was  presented  in  public 
last  April  at  the  TATRC  Exhibit  at 
the  American  Telemedicine 
Meeting  in  Orlando,  Florida,  April 
2003. 


Fig.  12.  Penelope  2,  in  November  2002.  This  was  the 
machine  in  the  demo  video  that  was  viewed  by  the  TATRC 
review  committee  for  our  original  proposal.  This  machine 
has  many  physical  improvements  compared  to  P-1. 


Fig.  11.  Penelope  1  at  time  of  the  original  proposal  to 
TATRC,  August  2002,  shortly  after  Dr.  Moses  visited 
Columbia  University. 
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Penelope  2.5 

Software  Improvements 

The  software  improvements  incorporated  in  Penelope  2.5  were  major  and  resulted  in  code  for  the 
motion  control  system  that  runs  approximately  ten  times  faster  than  the  code  we  had  when  we 

submitted  the  proposal.  This  new 
code  allows  the  robot  system  to 
run  very  nicely  on  a  laptop 
computer,  whereas  formerly  it 
took  a  rather  large  and  heavy  dual 
processor  system  to  run  the  code. 
The  faster,  more  efficient  code 
means  it  will  be  possible  to 
achieve  a  greater  number  of 
calculations  per  second  of  the 
update  of  the  motion  of  the  arm. 
As  was  explained  in  the  preceding 
section  on  the  Motion  System,  the 
calculations  of  the  motion  of  the 
arm  proceed  on  time-step  by  time- 
step  basis.  For  each  time-step,  a 
small  increment  of  the  position, 
velocity  and  acceleration  of  the 
arm  is  calculated  from  the 
instantaneous  forces  applied  to  the 
arm  at  the  beginning  of  the  time- 
step.  Smaller  time-steps  can 
produce  a  finer  grain  of  the  motion 
calculations,  but  require  more 
computing  cycles.  Faster,  more 
efficient  code  means  it  is  possible 
using  a  reasonable  sized  computer 
to  break  the  motion  down  into 
smaller  time-steps  while  still 
having  enough  computer  resources 
left  to  manage  the  other  tasks  of 
the  robot,  particularly  the  vision  system  which  is  also  fairly  demanding. 

Another  important  aspect  of  the  system  was  the  integration  of  the  camera  system  into  the  main 
body  of  the  robot  code.  Prior  to  this,  for  P-2,  the  code  for  the  running  of  the  camera  hardware 
was  run  as  a  separate  application  which  timeshared  with  the  main  application  that  handled  the 
image  processing,  motion  control  and  other  aspects  of  the  robot.  The  image  processing  (i.e. 
object  localization  and  identification)  in  the  main  application  required  as  input  the  output  of  the 
camera  hardware,  which  was  running  continuously,  producing  a  continuous  stream  of  image 
data.  The  output  of  the  camera  hardware  was  made  available  to  the  main  application  by  means 
of  a  shared  file  containing  the  camera  output  data.  The  operating  system  of  the  computer  was 
generally,  but  not  always,  able  to  handle  the  conflicts  that  sometimes  arose  when  the  camera 
hardware  was  trying  to  update  the  shared  data  file  and  the  image  processing  software  was  trying 
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Fig.  13.  Penelope  2.5,  delivering  a  Kelly  clamp  at  the 
TATRC  Exhibit,  April  2003.  This  machine  has  revised 
shoulder,  upper  arm  and  elbow  as  well  as  an  angled  gripper. 
There  are  also  extensive  software  improvements  over 
Penelope  2  which  allow  the  machine  to  run  on  a  laptop 
computer  and  to  be  relatively  impervious  to  variable  lighting 
conditions. 


to  read  in  the  file.  This  problem  was  exacerbated  at  frame  rates  over  5  fps  and  would 
occasionally  result  in  system  crashes.  A  major  breakthrough  was  development  of  software 
drivers  for  the  camera  hardware  that  could  be  called  directly  from  the  main  application.  This 
development  eliminated  the  inter-application  conflict,  removed  the  frame  speed  barrier  and 
increased  the  speed  of  the  overall  application.  Undoubtedly,  this  integration  was  the  key 
contributor  to  the  successful  demonstration  of  the  robot  at  the  TATRC  Exhibit  in  April,  2003. 

In  terms  of  the  motion  control  software,  we  completely  reorganized  and  rebuilt  the  physics-based 
simulation  which  is  at  the  heart  of  the  motion  control  system.  This  work  was  done  with  an  eye  to 
minimizing  the  number  of  software  “objects”  that  are  created  by  the  code.  In  the  older  code, 
software  objects  that  were  needed  to  run  the  simulation  were  re-created  at  each  time-step  of  the 
simulation.  This  made  the  code  easier  to  read  and  understand  but  ended  up  creating  an  enormous 
number  of  such  objects,  since  the  simulation  was  being  run  at  the  fairly  high  rate  of  fifty  steps 
per  second.  Minimizing  the  number  of  software  objects  created  during  each  time  step  was 
accomplished  by  re-using  these  objects.  This  made  the  code  somewhat  less  intuitive  to 
understand  but  the  performance  increase  was  very  impressive.  The  impact  of  the  performance 
increase  was  that  we  were  able  to  run  the  robot  code  on  a  laptop  with  approximately  half  the 
computing  power  as  the  machine  required  to  run  P-2’s  code.  As  explained  above  in  the  Motion 
System  section,  the  robot  is  represented  by  a  “model”  which  is  put  together  out  of  specified 
components  when  the  program  is  started  up.  These  components  are  the  “rigid  bodies”  which 
correspond  to  the  segments  of  the  arm.  The  software  revision  was  also  done  with  an  eye  to 
making  the  robot  models  independent  of  the  part  of  the  software  that  runs  the  simulation.  This 
makes  it  easier  to  change  the  model  in  order  to  mirror  the  robot’s  physical  structure. 

Other  changes  incorporated  in  the  P-2.5  code  were  useful  for  the  successful  demonstration  that 
we  had  at  the  TATRC  Exhibit.  These  were  changes  to  the  user  interface  that  allowed  much 
easier  and  accurate  calibration  of  the  camera  system  and  the  servomotors,  and  to  register  the 
camera  system  coordinates  with  the  coordinate  system  in  which  the  arm  moves.  The  camera 
system  user  interface  now  included  calibrating  out  the  effects  of  background  illumination  and 
shadows.  This  ability  to  calibrate  the  camera  system  “on-the-fly”  was  crucial  to  the  vision 
system’s  effective  performance  at  the  TATRC  Exhibit.  At  various  times,  during  the  exhibit,  a 
moving  speckled  pattern  of  light  played  across  the  transfer  zone.  We  are  able  to  calibrate  out  the 
color  of  that  light  such  that  the  vision  system  could  ignore  it  and  successfully  recognize  the 
instruments. 

In  general,  the  TATRC  Exhibit  was  good  test  for  the  robot’s  ability  to  perform  under  something 
approximating  real-world  conditions. 


Mechanical  Improvements 

As  mentioned,  the  shoulder,  upper  arm,  and  elbow  were  rebuilt  to  eliminate  structural 
weaknesses  of  the  P-2  design.  At  the  shoulder  level,  we  eliminated  the  “twist  “  degree  of 
freedom  servo,  since  this  capability  was  of  no  use  in  picking  up  instruments.  The  upper  arm  was 
revised  to  be  stiff er,  lighter  and  simpler  to  align.  Instead  of  the  box-truss  construction  of  the  P-2 
arm,  a  single  large  bore  carbon  fiber  tubing  was  used  for  the  forearm.  This  permitted  a 
simplification  of  the  elbow  joint,  resulting  in  a  joint  that  was  less  prone  to  distortion  under  stress 
and  also  lighter.  Another  mechanical  change  was  to  position  the  electromagnetic  gripper  at  a 
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20%  angle  offset  from  the  long  axis  of 
the  forearm.  The  change  in  gripper 
placement  was  subtle,  but  it  allowed 
the  robot  to  pick  up  instruments  more 
reliably  over  a  larger  area  of  the 
transfer  zone.  We  also  designed  and 
built  a  fiberglass  mounting  platform, 
which  integrates  the  arm,  the  camera  post,  the 
Mayo  Stand  and  the  transfer  zone.  This  fiberglass  platform  gave 
the  robot  a  more  polished  look  and  suggested  how  an  actual 
clinical  system  would  be  configured.  We  designated  this  system, 
with  the  software  improvements  described  above,  as  Penelope  2.5. 

Performance  Results  of  P-2.5 


Fig.  14.  Slightly  offset  gripper 
improved  pick-up  capability 
over  the  transfer  zone,  but  also 
suggested  need  for  an  actual 
“wrist". 


This  section  describes  the  performance  of  the  system  after  the  above  hardware  and  software 
improvements  were  made  and  incorporated.  It  does  not  take  into  account  the  results  of  the 
improvements  to  the  Visual  System,  as  this  work  is  being  integrated  at  the  present  time  into  P-3. 


The  P-2.5  system  performed  very  well  at  the  TATRC  exhibit  with  an  instrument  assortment  of 
Kelly  clamp,  Hopkins  clamp,  Metzenbaum  scissors  and  Adson-Brown  forceps.  It  successfully 
completed  several  hundred  instrument  retrievals  with  an  overall  success  rate  of  nearly  90%.  This 
success  rate  was  not  rigorously  determined  at  the  meeting,  but  was  subsequently  in  fact  shown  to 
be  correct  by  formal  testing  in  the  lab.  This  success  rate  is  the  combined  overall  success  rate  for 
the  entire  system,  working  on  four  instruments.  Individual  components  of  the  system  worked 
well.  The  vision  routines  worked  very  well,  in  the  setting  of  variable  lighting  conditions. 
Mechanical  accuracy  was  quite  good,  but  the  pick-up  success  rate  was  definitely  better  for 
instruments  dropped  near  the  center  of  the  transfer  zone.  The  voice  recognition  worked 
reasonably  well  with  the  use  of  a  headset  microphone  but  remains  an  area  which  needs 
improvement. 

Penelope  3 

When  we  returned  from  the  April  TATRC  Exhibit,  we  realized  as  the  result  of  demonstrating 
that  machine  to  many  people  that  it  would  have  to  be  much  faster.  It  was  clear  that  we  were  up 
against  the  mechanical  limits  of  the  servos  that  we  were  using,  both  from  the  speed  and  accuracy 
standpoint.  It  is  clear  also  that  overall  physical  layout  of  P  2.5  would  not  support  the  42  types  of 
instruments  found  in  the  Minor  Tray.  We  decided  at  that  time  that  the  path  to  take  to  achieve  a 
clinically  useable  machine  was  not  to  work  on  the  software  solution  via  visual  servoing  in 
improve  the  performance  of  the  servomotors,  but  to  completely  re-design  the  arm  from  the 
ground  up,  using  a  better  type  of  actuator-  steppers  motors.  The  new  actuators  would  solve  not 
only  the  accuracy  problem,  but  the  speed  problem,  which  could  never  be  overcome  with  the 
servos  of  P-2.  We  also  undertook  the  large  task  of  designing  a  full  clinical  machine.  This  was  a 
complex  process  that  was  undertaken  with  an  eye  to  producing  the  documentation  that  would  be 
necessary  to  apply  for  FDA  approval.  To  this  end,  we  instituted  a  Quality  System  with  Design 
Controls  for  the  clinical  robot  development  program. 

In  the  past  six  months,  we  have  produced  a  completely  redesigned  arm,  which  is  based  on  a 

19 


completely  redesigned  physical  layout  of  the  entire  system.  The  design  goal  of  the  new  system 
is  to  minimize  instrument  transfer  time  while  providing  the  physical  structure  to  be  able  to 
accommodate  the  42  instrument  types  found  on  the  conventional  Minor  Tray.  The  new  system  is 
designated  Penelope  3  and  is  being  completed  at  this  time.  It  incorporates  the  results  of  the 
work  done  under  our  TATRC  contract  and  builds  on  these  results  to  approach  our  ultimate  goal 
of  clinical  usability. 

Engineering  Requirements 

The  engineering  requirements  of  our  clinical  grade  robot,  Penelope  3,  have  been  codified  in  our 
Requirements  Specification  Document  (RSD).  The  working  draft  of  the  RSD  contains  over  100 
requirements  covering  all  aspects  of  the  robot’s  functionality.  In  developing  the  RSD,  we 
gathered  user  inputs  from  clinical  staff  at  the  Allen  Pavilion  of  the  New  York  Presbyterian 
Hospital.  The  top-level  requirement  is  that  the  robot  in  no  way  impede  the  safe  and  expeditious 
completion  of  the  operation.  There  are  many  other  specific  requirements  that  support  this  overall 
requirement.  The  key  items  are  summarized  as  follows: 

Instrument  Handling:  The  robot  has  the  physical  architecture  to  handle  all  the  instrument 
types  on  the  Minor  Surgical  Tray.  The  physical  architecture  includes  the  following 
instrument  holding  surfaces:  Mayo  stand,  transfer  zone,  staging  zone  and  back  tray. 

Sterile  Draping:  The  robot  can  be  prepared  for  sterile  use  by  the  circulating  nurse  who 
applies  specially  designed  sterile  drapes.  We  have  designed  proprietary  draping  fixtures 
and  tools  that  cover  the  various  working  surfaces  such  as  the  Mayo  stand  and  the  transfer 
zone. 

Vision  System  Performance:  The  requirement  is  to  be  able  to  recognize  all  the 
instruments  on  the  minor  surgical  set,  about  42  types.  As  a  result  of  the  TATRC  work, 
the  computer  vision  system  has  been  extended  to  be  able  to  recognize  and  distinguish 
twelve  instruments  with  over  98%  accuracy.  Just  as  importantly,  the  general  methods  for 
extending  the  vision  routines  to  handle  more  instruments  have  been  developed. 

Speed  and  Accuracy  of  Performance:  Our  engineering  analysis  of  the  completely 
redesigned  arm  indicates  that  the  arm  will  be  able  to  meet  or  exceed  requirements  for 
speed  and  accuracy. 

Safety:  Basic  patient  safety  has  been  designed  into  the  physical  architecture  of  the  robot. 
The  robot  is  physically  not  able  to  impact  its  arm  on  the  patient  or  to  even  drop  an 
instrument  onto  the  patient.  The  robot  will  not  directly  handle  sharps  (scalpels  and 
loaded  sutures).  As  recommended  by  OSHA  for  human  scrub  technicians,  these  items 
will  be  made  available  to  the  surgeon  in  specially  designed  (proprietary  and  disposable) 
scalpel  and  suture  holders  that  can  be  picked  up  by  the  robot’s  electromagnetic  gripper. 

Penelope  3,  albeit  with  the  arm  of  Penelope  2.5,  was  presented  at  TATRC  DAY  during  the 
Medicine  Meets  Virtual  Reality  Meeting  in  Newport  Beach,  CA,  January  2004.  The  robot  was 
recognizing  twelve  instruments  very  well  in  real  world  conditions,  and  was  moving  them  fairly 
well,  given  the  limitations  of  the  older  2.5  arm.  As  a  demonstration  of  the  versatility  of  the 
vision  and  motion  system,  the  robot  was  programmed  to  visually  recognize  a  piece  of  paper  upon 
which  a  Tic-Tac-Toe  playing  grid  was  drawn,  and  then  to  inquire  “How  about  a  game  of  Tic- 
Tac-Toe?”  For  the  game,  the  robot  visually  recognizes  the  positions  of  color  coded  “X”  markers 
placed  by  the  human  player  and  then  places  “O”  markers  (steel  washers)  onto  the  board  for  its 
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own  moves. 


As  part  of  our  exhibit,  the  new  arm  was  on  display,  but  was  not  at  that  time  functional. 


Fig.  15.  Penelope  3  at  TATRC  DAY,  January,  2004  in  Long 
Beach,  CA  as  part  of  the  Medicine  Meets  Virtual  Reality 
meeting.  This  robot  has  the  physical  architecture  to  support 
many  more  instruments  than  previous  versions. 
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Related  Work  Contributing  to  Clinical  Goal 

NSF  SBIR  Phase  I  Work 

During  the  past  six  months,  we  also  successfully  completed  our  Phase  I  Technical  Project  for  the 
National  Science  Foundation  SBIR  award  that  we  received.  The  Phase  I  research  objective  was 
to  determine  the  feasibility  of  using  artificial  intelligence  and  statistical  techniques  to  predict  the 
surgeon’s  instrument  requests.  This  will  allow  the  robot  to  keep  one  step  ahead  of  the  surgeon, 
much  like  an  experienced  human  scrub  technician.  With  this  predictive  capability  the  robot  can 
decide  how  to  organize  its  limited  storage  space  to  keep  those  instruments  likely  to  be  needed 
soon  closer  to  the  surgical  field.  This  will  greatly  improve  the  responsiveness  of  the  device,  a 
critical  factor  in  achieving  clinical  acceptance  and  ultimately  commercial  viability. 

These  results  will  provide  a  crucial  framework  for  the  broader  task  of  creating  a  cognitive 
architecture  for  the  robot.  This  architecture  will  control  the  robot’s  behavior  and  enable  it  to 
adapt  to  the  ever-changing  environment  in  the  OR.  A  reliable  instrument  prediction  capability 
will  allow  the  robot  to  exhibit  proactive  behavior,  as  opposed  to  merely  reacting  to  explicit 
commands.  This  is  a  fundamental  distinction,  separating  traditional  devices  from  truly 
autonomous  robots.  We  believe  that  this  autonomy  is  an  essential  advance  for  robotics  in  the 
OR. 

In  the  Phase  I  work,  we  recorded  from  actual  surgeries  a  database  of  over  50  surgical  procedures, 
cataloging  over  2000  individual  instrument  requests.  We  then  used  this  time  series  data  to  train 
and  evaluate  a  prediction  algorithm.  Our  best  algorithm  was  a  modified  N-gram  sequence 
matcher.  At  each  point  in  the  surgical  procedure,  the  algorithm  produces  a  prediction  score  for 
every  instrument  type.  The  score  for  an  instrument  is  the  likelihood  of  its  being  the  next 
instrument  selected,  given  a  particular  prior  sequence  of  instruments.  The  instrument  with  the 
highest  score  would  be  the  one  which  has  the  greatest  chance  of  being  selected  next.  In  doing 
our  Phase  I  work,  we  had  to  consider  carefully  how  these  scores  were  going  to  be  used  to  deliver 
on  our  objective  of  improving  the  robot’s  overall  performance.  Due  to  the  variable  nature  of 
surgery,  relying  on  a  prediction  of  one  instrument  could  commit  the  robot  to  an  inappropriate 
action,  i.e.  presenting  the  surgeon  with  the  wrong  instrument.  However,  we  found  that  by  taking 
into  account  the  set  of  most  likely  instruments,  we  could  significantly  improve  overall 
performance.  In  order  to  use  the  information  provided  by  predicting  the  set  of  likeliest 
instruments,  we  developed  the  concept  of  the  robot  as  an  instrument  server,  analogous  to  a 
computer  file  server.  In  the  instrument  server  concept,  the  robot’s  Job  is  to  keep  frequently 
requested  instruments  on  the  “fast”  caches  (the  ones  closer  to  the  surgeon)  and  less  frequently 
used  instruments  on  the  slower  caches  (the  ones  further  away  from  the  surgeon).  The  robot  uses 
the  prediction  scores  to  decide  which  instruments  to  keep  on  the  various  caches.  This  concept 
elegantly  integrates  the  results  of  the  prediction  algorithm  with  a  physical  architecture  for  the 
robot  that  is  optimized  for  responsive  instrument  delivery.  It  is  also  very  reminiscent  of  the 
situational  awareness  of  the  experienced  scrub  person,  who  proactively  manages  the  instruments 
to  stay  in  step  with  what  the  surgeon  is  doing. 
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Fig.  16.  An  overhead  view  of  the  robotic  scrub  technician  (shown  without  sterile  draping  for 
clarity).  Note  the  various  surfaces  on  which  the  robot  stores  instruments.  Some  are  close  to 
the  surgeon  for  quick  delivery,  while  others  require  the  robot  to  spin  around  taking  more  time. 

We  proved  this  performance  increase  by  means  of  a  simulation  in  which  the  prediction 
algorithm,  after  being  trained,  was  run  against  a  test  case  that  was  not  part  of  the  training  set. 
The  basis  for  comparison  was  a  baseline  strategy  of  keeping  the  twelve  overall  most  commonly 
used  instruments  on  the  Mayo  stand,  which  is  a  “fast”  cache  close  to  the  surgeon.  In  the 
simulation,  the  prediction  scores  are  used  to  decide  which  instruments  to  move  forward  to  fast 
caches  including  the  transfer  zone  and 
which  to  move  back  to  “slower”  caches. 

The  right  moves  will  keep  the  requisite 
instruments  close  to  the  surgeon, 
minimizing  our  primary  metric,  average 
instrument  delivery  time.  We  found  that 
our  algorithm  could  recommend 
favorable  moves  88%  of  the  time.  This 
resulted  in  a  51%  decrease  in  the  average 
instrument  delivery  time  as  compared  to 
our  baseline  strategy.  Moreover,  when 
trained  on  each  surgical  procedure  from 
the  data  set  in  chronological  order,  the 
algorithm’s  performance  improved  over 
time  (fig.  17). 

In  Phase  I,  we  showed  how  the  output  of 
the  prediction  engine  could  be  used  by  a  simple  rule-based  system  to  improve  instrument 
delivery  time.  This  is  a  first  and  critical  step  towards  a  rule-based  cognitive  architecture.  The 
purpose  of  the  cognitive  architecture  is  to  give  the  robot  situational  awareness.  In  our  work, 
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Surgical  procedures  in  chronological  order 


Fig.  17.  The  chronological  learning  curve 
showing  the  decrease  over  time  in  instrument 
delivery  time  as  the  prediction  algorithm  is 
trained  on  more  and  more  data. 


‘situational  awareness’  means  a  multifaceted  knowledge  of  the  status  of  the  procedure  so  that  it 
can  take  the  appropriate  actions  that  facilitate  its  completion.  Anticipating  instrument  requests 
provides  a  certain  amount  of  situational  awareness.  A  more  complete  situational  awareness 
enables  the  robot  to  deal  with  routine  as  well  as  off-nominal  conditions  such  as  errors  and 
emergencies.  In  addition  to  the  Prediction  Engine,  we  will  use  the  outputs  of  other  systems,  such 
as  the  vision  system,  as  inputs  for  rules  of  behavior.  This  richer  set  of  rules  will  bring  us  closer  to 
our  overall  goal  of  a  robotic  scrub  technician  which  can  match  the  overall  functionality  of  a 
human  scrub  technician  in  the  operating  room. 


Fig.  18.  Functional  overview  of  what  a  cognitive 
architecture  does  for  Penelope. 


A  cognitive  architecture  (CA)  is  a  description  of  a  mind,  uniting  the  sensory  and  reasoning 
subsystems  and  processes  that  together  provide  for  cognition.  The  design  of  a  CA  dictates  how 
these  subsystems  are  functionally  interrelated.  Many  CA’s  of  widely  varying  design  have  been 
created  during  the  past  23  years,  primarily  for  the  benefit  of  two  fields:  cognitive  psychology  and 
artificial  intelligence.  Most  of  the  established  CA’s  have  a  set  of  top-level  elements  in  common. 
Multiple  sensory  subsystems  provide  inputs  that  are  considered  together  before  the  system 
produces  behavior  through  one  or  more  physical  output  subsystems.  There  is  a  knowledge  base, 
a  set  of  things  it  knows  to  be  true  without  having  to  reestablish  them  every  time  a  decision  must 
be  made.  This  corresponds  roughly  to  our  human  memory,  and  many  CA’s  divide  it  into  long¬ 
term  and  working  memory.  Reasoning  ability  is  provided  by  the  production  system,  a  set  of  if- 


24 


then  rules  or  ‘productions’  that  together  determine  the  mapping  from  inputs  to  output  behaviors. 
Each  rule  has  several  conditions  and  one  or  more  output  actions.  Every  time  a  new  piece  of 
information  comes  in  from  an  input,  the  rules  are  reevaluated  to  update  the  system’s 
understanding  of  the  situation,  and  new  actions  may  be  taken.  The  advantage  afforded  by  a  CA 
is  the  ability  to  balance  and  consider  the  multiple  inputs  in  producing  an  overall  perception  and 
to  reconcile  discrepancies  in  the  information  from  the  inputs.  These  abilities  are  vital  in  a  system 
that  must  understand  and  participate  in  the  complex  OR  environment  without  impeding  the 
operation.  The  design  of  our  cognitive  architecture  will  draw  on  that  of  previously  developed 
systems,  particularly  one  called  “Soar”.  Soar  is  a  very  capable  system  developed  for  artificial 
intelligence  and  has  been  fielded  in  applications  ranging  from  intelligent  game  opponents  to 
fighter  plane  navigation. 

Validating  the  Cognitive  Architecture 

To  make  a  distinction  between  verification  and  validation,  it  is  often  said  that  “verification  is 
building  the  system  right;  validation  is  building  the  right  system.”  The  ‘right  system’  can  be  a 
difficult  concept  to  pin  down  but  we  feel  that  the  process  outlined  will  over  time  result  in  the 
clinically  desired  behavior.  To  validate  the  utility  of  the  cognitive  architecture,  we  must  devise  a 
suitable  means  of  evaluating  its  performance  as  a  whole.  Testing  the  entire  system  requires  a 
realistic  setting  where  the  robot  will  experience  the  full  range  of  interactions  with  the  OR 
environment.  The  main  activity  will  be  a  repetitive  process  of  validation  and  modification,  until 
acceptable  clinical  behavior  has  been  achieved. 


Key  Research  Accomplishments 

The  vision  routines  are  handling  twelve  instruments  with  an  accuracy  rate  of  over  98%. 
The  motion  system  is  currently  performing  at  a  94%  accuracy  rate. 

The  completely  re-engineered  stepper  motor  arm  has  been  built  and  is  nearing  operational 
status.  It  was  made  available  for  inspection  TATRC  Day,  January  14,  2004.  This  new 
arm  will  take  care  of  accuracy  and  speed  issues. 


Reportable  Outcomes 

A  complete  design  of  a  full  featured,  realistic  clinical  grade  system,  Penelope  3,  with 
supporting  documentation  needed  for  FDA  application,  is  well  underway. 

A  physical  implementation  of  Penelope  3  was  available  for  demonstration  at  TATRC 
Day,  January  14,  2004.  This  is  not  the  final  implementation,  since  the  new  arm  is  not 
integrated  yet,  but  it  convincingly  shows  the  key  design  features  needed  for  clinical  use. 
As  a  result  of  all  of  this  work,  we  are  in  a  strong  position  to  apply  for  our  Phase  II  SBIR 
funding.  Our  Phase  II  SBIR  work  is  about  developing  a  cognitive  architecture  for  P-3. 
The  cognitive  architecture  will  extend  the  instrument  prediction  software  that  we 
developed  under  the  SBIR  Phase  I  work.  The  TATRC  sponsored  research  is  entirely 
supportive  of  this  Phase  II  application,  since  this  work  demonstrates  the  software  and 
hardware  infrastructure  needed  to  make  the  cognitive  system  and  the  entire  robot  reality. 
As  a  result  of  this  work,  we  have  also  won  matching  funds  from  the  New  York  State 
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Office  of  Science,  Technology  and  Academic  Research  (NYSTAR). 

The  work  has  also  led  to  a  commitment  from  Dr.  Herbert  Pardes,  President  and  CEO  of 
the  New  York-Presbyterian  Hospital  to  be  the  first  clinical  site. 

Progress  with  the  development  has  attracted  the  attention  of  local  and  national  media, 
including  the  New  York  Times.  The  innovation  and  societal  value  of  the  robot  are 
apparent  and  are  well  received  by  health  care  professionals,  hospital  administrators,  and 
the  public. 

Technical  success  has  strengthened  our  position  to  reach  out  to  investors.  A  business 
plan  for  Robotic  Surgical  Tech,  Inc.  has  attracted  the  support  of  the  Columbia  Business 
School  faculty  and  has  been  accepted  into  the  Greenhouse  Competition.  The 
Greenhouse  competition  means  that  the  plan  will  be  vetted  by  Business  School  faculty 
and  if  successful,  will  be  awarded  seed  money  from  the  Lang  Fund,  an  in-house  fund  to 
encourage  entrepreneurial  activity. 


Conclusions 

The  TATRC  research  contract  has  made  possible  significant  improvements  in  the  technology 
base  of  the  robotic  scrub  technician.  The  TATRC  support  has  also  been  very  useful  in  helping  us 
to  gain  other  types  of  support,  including  National  Science  Foundation  support  and  matching 
funds  from  New  York  State.  We  believe  that  all  of  the  benefits  of  all  this  work,  technological 
and  other,  are  contributing  to  the  overall  goal  of  a  clinically  successful  robotic  scrub  technician. 
We  are  very  thankful  to  TATRC  for  giving  us  this  opportunity.  We  will  do  whatever  possible  to 
bring  credit  to  TATRC  by  producing  a  product  that  will  advance  the  art  of  surgical  care  in  both 
civilian  and  military  settings. 

Report  respectfully  submitted  by: 

Michael  R.  Treat  MS,  MD 
February  2004 
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Fig.  19.  The  Penelope  3  prototype  will  be  able  to  handle  clinical  cases.  This  robot  will  be 
the  test-bed  for  the  proposed  cognitive  architecture  to  validate  overall  clinical  utility  of  the 
entire  system,  hardware  and  software,  including  advances  made  under  the  TATRC  contract. 
This  version  is  shown  with  the  new  stepper  motor  arm. 


Fig.  20.  Penelope  Development  Team:  from  left  to  right,  Michael  R.  Treat, 
Russell  L.  Baker,  D.  Michael  Brady  and  Jay  A.  Klein. 
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Appendices 


Sire  jNW  Hork  Simes 


iNursing  Shortage  Forces  Hospitals  to  Cope  Creatively 


j  By  LAURIE  TARKAN 

|[  Banter  In  the  hospital  operating  room 
?  may  be  less  lively  If  the  surgeon's  assistant 
^•Is  a  robot  rather  than  a  nurse,  but  that  Is  of 
vno  concern  to  the  inventor  of  Penelope.  With 
'  a  six-year^oId  nursing  shortage  showing  no 
signs  of  easing,  Dr,  Michael  R.  Treat  of  the 
;  polumbla-Presbytcrlan  Center  in  Manhat- 
I' Ian  is  hoping  that  his  one-armed  robot  will 
j;  replace  the  nurse  who  hands  the  surgeon  the 
I  instruments,  freeing  the  nurse  to  give  post¬ 
il  operative  care, 

f  ■  Other  robots  already  ferry  medications 
j|  and  supplies  around  hospUaLs. 
r  With  mechanical  help,  flexible  shifts  and 
[online  auctions  of  shifts,  hospltols  arc  sur- 
1;  passing  the  creative  In  dealing  with  the 
nursing  shortage  that  experts  predict  will 
If  worsen  In  a  decade  or  two. 

F  The  pressure  on  hospitals  to  attract  and 
[retain  nurses  continues  to  grow,  largely 
I  because  of  a  mounting  body  of  evidence  that 
1;  being  short  staffed  compounds  the  rate  of 
\  medical  errors  and  deaths, 

I  On  Jan.  1,  California  became  the  first 
stale  to  mandate  specific  nurse-to-patleni 
I  ratios.  Hospitals  there  have  been  scram- 
f :  bllng  to  meet  the  deadline.  ' 

(y:  Around  the  country,  using  various  strat- 
f  egles,  some  are  beginning  to  see  their  ef- 
I  lorts  succeed,  leading  to  lower  vacancy 
'  rates  In  nursing  Jobs,  lower  turnover  and 
lower  mortality  rates  for  patients.  In  addi¬ 
tion,  hospitals  are  seeing  higher  ratings  of 
satisfaction  among  nurses  and  greater  sat- 
{  isfaction  among  patients. 

In  efforts  to  keep  health  care  costs  down 
In  the  1980's  and  early  90’s,  hospitals  elimi¬ 
nated  nursing  positions  and  tried  to  increase 
efficiency,  but  often  at  the  expense  of 
nurses’  working  conditions,  experts  con- 
tend,  decreasing  their  flexibility,  int^asing 
y  their  workload  and  reducing  their  proles  In 
decision  making. 

About  13  percent  of  nursing  positions  na- 


That  relationship  is  not  the  only  problem 
that  has  to  be  noticed.  A  report  just  re¬ 
leased  by  the  Institute  of  Medicine  of  the 
National  Academies  found  that  nurses’ 
working  conditions  were  contributing  sig¬ 
nificantly  to  medical  errors. 

High  patlent-to- nurse  ratios,  fatigue  on 
long  shifts  and  mandatory  overtime,  a  luck 
of  experienced  staff,  and  inadequate  time  to 
monitor  patients  have  been  associated  wltli 
poor  medical  results  and  higher  death  rates 
for  patients,  the  report  said. 

“Nurses  can  commit  errors,  and  they 
also  play  a  crucial  role  in  protecting  pa¬ 
tients  from  errors,”  said  Dr.  Donald  M. 
Steinwachs,  who  led  the  panel  that  Issued 
the  report. 

According  to  a  paper  by  the  Joint  Com¬ 
mission  on  the  Accreditation  of  Healthcare 
Organizations,  low  numbers  for  nursing 
staffs  was  a  factor  in  19  percent  of  medical 
errors  resulting  in  deaths  or  serious  inju¬ 
ries  In  hospitals.  Nurses’  inadequate  orien¬ 
tation  and  training  were  cited  as  factors  in 
58  percent  of  serious  errors. 

A  study  by  Dr.  Aiken  found  that  patients 
scheduled  (or  routine  surgery  were  31  per- 
Dr.  Michael  R.  Treat  hopes  his  one-armed  robot  will  someday  replace  the  nurse  who  cent  more  likely  to  die  In  a  hospital  with  a 
hands  the  surgeon  the  instruments,  freeing  the  nurse  to  give  postoperative  care.  patlent-to-nurse  ratio  of  el^t  to  one  fhatUn 

_ Z _ ! _ t - r - 1 - : -  a  hospital  wltli  a  ratio  of  four  to  one.  The 

.  .  .  .  L  .  I.  .  study  was  published  last  year  in  The  Jour- 

tionwlde  are  vacant,  the  American  Hospital  to  be  less  particular  about  their  prospective  American  Medical  Association. 

Association  reports.  Experts  predict  that  employees,  said  Dr.  Linda  Aiken,  a  profes-  A  majority  of  nurses  say  they  believe  that 
the  rate  will  Increase  to  20  percent  by  2015.  sor  at  the  Unlversltyof  Pennsylvania  School  ^.jjnnot  do  their  jobs  as  well  wider 

“Every  hospital  wants  to  hire  more*  of  Nursing,  who  is  a  leading  researcher  on  current  working  conditions,  a  2001 

nurses  and  improve  the  working  condl-  the  shortage.  survey  by  the  .American  Nurses  Association 

tions,”  said  Amy  Lee,  spokeswoman  for  the  “High  turnover  rate,”  Dr,  Aiken  said,  “is  shows. 

American  Hospital  Association,  “but  hospl-  a  very  big  problem  and  potentially  leads  The  survey  found  that  75  percent  of 

tals  are  in  fragile  financial  states,  and  some  hospitals  to  be  desperate  to  try  to  get  In  nurses  said  the  quality  of  nursing  care  at 

don’t  have  financial  capital  to  do  it."  sufficient  numbers  of  nurses  to  keep  their  their  medical  centers  had  declined  In  the 

Tlie  nursing  shortage  may  have  helped  services  open."  prior  two  years.  More  thsm  40  percent  said  ^ 

Charles  Cullen,  the  nurse  who  the  police  say  Many  hospitals  have  dealt  with  the  short-  ^cy  would  not  feel  comfortable  having  a 

has  admitted  kllli  ig  30  to  40  patients,  to  age  by  requiring  overtime  in  understaffed 
move  from  one  hof  ~  [  ~  ~ 

jenKjyimdPennsy  jaouary  6,  2004  atticle  from  the  New  York  Times 
hospitals  less  cons]  featuplng  Penelope.  For  the  full  text  of  the  article  visit 
our  website  at 

http://www.roboticsurgicaltech.com/NYTimesArticle.pdf 
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EMAIL 
June  17, 2003 


Michael  R.  Treat,  M.D. 
mt23@columbia.edu 


Dear  Michael, 

Thanks  very  much  for  the  information  on  the  Robotics  Scrub  Tech  project.  I  believe  you  said  a 
clinical  debut  was  possible  in  2004.  I  gather  that  you  are  working  closely  with  David  Liss.  It 
may  well  be  that  you  cannot  be  anymore  specific  about  the  date  for  a  rollout,  but  I  am  excited 
about  the  possibility  of  rolling  it  out  first  at  NewYork-Presbyterian  and  am  very  pleased  with  the 
work  you  are  doing. 

Let  me  know  what  you  think  and  keep  me  posted. 


All  the  best. 


Sincerely, 

Herbert  Pardes,  M.D. 
HP/ys 


Letter  of  support  from  Dr.  Herbert 
Pardes,  President  and  CEO  of  New 
York-Presbyterian  Hospital,  stating  his 
interest  in  being  the  first  customer  for 
the  robotic  scrub  technician. 
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ABSTRACT 

The  Penelope  System  Requirements  Specification  Document,  PS-RSD-090803,  is 
provided  in  accordance  with  the  Penelope  System  Project  Plan,  PS-PP-<TBD>.  This 
document  is  prepared  by  Robotic  Surgical  Tech,  Inc.  It  contains  the  requirements  for 
development  of  the  Penelope  System  Robotic  Scrub  Technician. 
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SECTION  1,  INTRODUCTION 

1 . 1  SYSTEM  OVERVIEW 

The  Penelope  System  Robotic  Scrub  Technician  is  designed  to  perform  the  functions  of 
the  scrub  nurse  or  technician  in  the  Operating  Room  (OR).  The  robot  uses  voice 
recognition  to  respond  to  a  surgeon’s  request  for  an  instrument.  A  manipulator  arm  with 
an  electromagnetic  gripper  delivers  the  instrument  from  the  Mayo  stand  to  the  surgeon. 
When  the  surgeon  is  finished  with  the  instrument  he/she  places  it  on  the  transfer  zone. 
The  vision  system  equipped  with  a  digital  camera  locates  and  identifies  the  instrument 
and  provides  coordinates  to  the  motion  control  system  to  direct  the  robot’s  arm  to  retrieve 
the  instrument. 

In  general  terms,  the  objectives  of  the  Penelope  System  are  to: 

A.  Listen  for  instrument  requests  from  the  surgeon  or,  more 
generally,  any  scrubbed  instrument  user  (SIU). 

B.  Deliver  requested  instruments  from  either  the  Mayo  stand  or  the 
back  tray  to  either  the  SIU  or  the  transfer  zone. 

C.  Scan  the  transfer  zone  for  relinquished  instruments. 

D.  Move  relinquished  instruments  from  the  transfer  zone  to  the 
Mayo  stand. 

E.  Give  a  verbal  instrument  count  when  requested. 

1.2  DOCUMENT  OVERVIEW 

This  document  defines  the  design  requirements  for  the  Penelope  System  Robotic  Scrub 
Technician.  All  requirements  for  the  system  are  defined  herein.  All  requirements 
defined  in  this  document  will  be  mapped  to  one  or  more  system  components  documented 
in  the  System  Design  Document  (PS-SDD-TBD).  Furthermore,  during  system  validation 
testing,  each  requirement  defined  herein  will  be  tested  for  compliance  as  per  the 
Verification  And  Validation  Plan  (PS-VV-TBD).  Successful  completion  of  the  Penelope 
System  Robotic  Scrub  Technician  will  be  defined  as  the  successful  validation  of  all  of  the 
requirements  defined  in  this  document. 

These  design  requirements  will  be  broken  down  into  three  categories:  functional, 
performance,  and  interface.  Functional  requirements  define  the  operational  capabilities 
of  the  system.  They  describe  the  high-level  tasks  the  system  shall  perform.  Performance 
requirements  define  the  set  of  quantitative,  measurable  parameters  within  which  the 
system  must  operate  to  be  effective  and  safe.  They  include  parameters  describing  the 
performance  of  the  robot  and  the  environmental  conditions  under  which  the  robot  must 
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be  operated.  Interface  requirements  define  all  interdependencies  between  the  Penelope 
System  and  external  systems.  This  includes  any  interactions  with  human  operators. 


1.3  USAGE  LIMITATIONS 

The  term  scrubbed  instrument  user,  or  SIU,  is  used  to  refer  to  any  OR  team  member  who 
is  sterile  and  can  manipulate  instruments.  Any  requirement  relating  to  the  transfer  of 
instruments  to  or  from  a  human  is  restricted  to  SIUs  in  accordance  with  the  Proper  Use 
Procedures  (<document  number  TBD>)  for  the  system.  The  Penelope  System  is 
specifically  not  required  to  distinguish  between  SIUs  and  nonsterile  personnel  in  any 
way.  Any  attempt  by  nonsterile  personnel  to  access  the  instrument  transfer  capabilities  of 
the  Penelope  System  is  considered  improper  use  of  the  system. 

All  surgical  instruments  introduced  into  the  Penelope  System’s  instrument  set  must  be 
sterile  in  accordance  with  the  Proper  Use  Procedures  (<document  number  TBD>)  for  the 
system.  The  Penelope  System  is  specifically  not  required  to  distinguish  between  sterile 
and  nonsterile  instruments.  Any  attempt  to  introduce  nonsterile  instruments  into  the 
Penelope  System  is  considered  improper  use  of  the  system. 

Any  command  input  to  the  system,  such  as  a  voice  command,  is  restricted  to  authorized 
OR  team  members  in  accordance  with  the  Proper  Use  Procedures  (<document  number 
TBD>)  for  the  system.  The  Penelope  System  is  specifically  not  required  to  distinguish 
between  command  inputs  from  authorized  OR  team  members  and  unauthorized 
personnel.  Any  attempt  by  unauthorized  personnel  to  issue  commands  to  the  Penelope 
System  is  considered  improper  use  of  the  system. 

All  requirements  relating  to  the  manipulation  or  processing  of  surgical  instruments  in  any 
way  is  restricted  to  only  those  instruments  of  a  type  listed  in  requirement  3. 1.3.1, 
“Supported  Instrument  Types.” 


<TBD:  more  about  what  we  do  not  have  the  ability  to  know.  This  is  the  garbage-in 
garbage-out  section. > 
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SECTION  2,  APPLICABLE  DOCUMENTS  LIST 
2. 1  APPLICABLE  DOCUMENTS 

The  following  documents  form  a  part  of  these  requirements  to  the  extent  specified  herein. 
RST: 


PS-SG-090803 

Penelope  System  Glossary 

PS-PP-<TBD> 

Penelope  System  Project  Plan 

PS-SDD-<TBD> 

Penelope  System  Design  Document 

<TBD> 

<TBD> 
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SECTION  3,  ENGINEERING  REQUIREMENTS 

3. 1  FUNCTIONAL  REQUIREMENTS 

3.1.1  Start  Up  And  Shut  Down  Requirements 

The  following  subparagraphs  list  all  functional  requirements  related  to  the  processes  of 
starting  up  and  shutting  down  the  Penelope  System. 

3 . 1 . 1 . 1  Power_Up_Command 

Upon  receipt  of  a  Power_Up_Command,  the  Penelope  System  shall  perform  the 
following  operations  in  order. 

1.  Issue  a  System_Start_Up_Command. 

2.  Determine  whether  the  robotic  arm  is  properly  positioned  in  its  designated  initial 
position. 

3.  If  the  arm  is  not  properly  positioned,  announce  “<TBD:  warning>”  and  then  issue 
a  System_Shut_Down_Command. 

4.  Otherwise,  move  the  robotic  arm  to  its  home  position  and  begin  command 
processing. 

3. 1.1. 2  System_Start_Up_Command 

Upon  receipt  of  a  System_Start_Up_Command,  the  Penelope  System  shall  perform  the 
following  operations  in  order. 

1 .  Supply  power  to  all  system  components,  including  the  microcontroller,  the  robotic 
arm,  and  System  Control  Processor. 

2.  Wait  for  the  System  Control  Software  to  issue  a  System_Started_Command, 
indicating  that  the  System  Control  Software  has  been  successfully  initialized. 

3.  If,  after  <TBD:  mmber>  seconds,  no  System_Started_Command  is  issued, 
remove  power  to  all  system  components,  including  the  microcontroller  executing 
this  procedure. 

4.  Otherwise,  continue. 

3. 1.1. 3  Power_Down_Command 

Upon  receipt  of  a  Power_Down_Command,  the  Penelope  System  shall  perform  the 
following  operations  in  order. 
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1.  Move  the  end  effector  and  any  instrument  it  may  be  holding  to  a  resting  position 
on  the  instrument  platform,  such  that  the  robotic  arm  will  not  fall  when  power  is 
removed. 

2.  Issue  a  System_Shut_Down_Command. 

3. 1.1. 4  System_Shut_Down_Command 

Upon  receipt  of  a  System_Shut_Down_Command,  the  Penelope  System  shall  perform 
the  following  operations  in  order. 

1 .  Remove  power  to  the  robotic  arm. 

2.  Issue  a  System_Halt_Command  to  the  System  Control  Software  instructing  it  to 
cease  execution  and  halt  the  System  Control  Processor. 

3.  Wait  for  the  System  Control  Processor  to  shut  down. 

4.  If,  after  <TBD:  mmber>  seconds,  the  System  Control  Processor  has  not  shut 
down,  continue  to  the  next  step. 

5.  Remove  power  to  all  remaining  system  components,  including  the  System 
Control  Processor  and  the  microcontroller  executing  this  procedure. 

3. 1.1. 5  Halt_Command 

Upon  receipt  of  a  Halt_Command,  the  Penelope  System  shall  stop  any  movement  of  the 
robotic  arm,  terminate  the  processing  of  any  other  commands  in  progress,  and  disable  any 
command  input  other  than  the  Power_Down_Command  or  the  Resume_Command.  If 
upon  receipt  of  a  Halt_Conimand  the  end  effector  is  currently  holding  an  instrument,  the 
Penelope  System  shall  continue  to  hold  that  instrument. 

3. 1.1. 6  Resume_Command 

Upon  receipt  of  a  Resume_Command,  the  Penelope  System  shall  enable  movement  of  the 
robotic  arm,  issue  a  Cancel_Command,  and  enable  command  input  for  all  commands. 
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3.1.2  Instrument  Request  Requirements 

The  following  subparagraphs  list  all  functional  requirements  related  to  the  handling  of 
requests  for  instruments  by  SIUs. 

3. 1 .2. 1  Gestural  Instrument  Requests 

The  Penelope  System  shall  detect  and  identify  gestural  instrument  requests  from  SIUs. 
Upon  identification  of  a  gestural  instrument  request,  the  Penelope  System  shall  issue  an 
Instrument_Request_Command.  <TBD:  more  detail  here  about  what  kinds  of  gestures 
and  so  on.> 

3. 1 .2.2  Multiple  Instrument  Requests 

The  Penelope  System  shall  be  capable  of  handling  more  than  one 

Instrument_Request_Command  at  a  time.  If  during  the  processing  of  one  such  command 
another  is  issued,  the  Penelope  System  shall  store  the  pending  requests  for  later 
processing.  These  pending  instrument  requests  are  not  required  to  be  processed  in  any 
particular  order. 

3. 1 .2.3  Instrument_Request_Command:  Instrument  Is  Not  Available 

Upon  receipt  of  an  Instrument_Request_Command,  if  all  instruments  of  the  requested 
type  are  in  use,  the  Penelope  System  shall  announce  “<TBD:  warning>".  This 
requirement  defines  a  special  condition  for  the  Instrument_Request_Command. 

3. 1 .2.4  Instrument_Request_Command:  Holding  An  Instrument 

Upon  receipt  of  an  Instrument_Request_Command,  if  the  end  effector  is  currently 
holding  an  instrument,  the  Penelope  System  shall  check  to  see  if  the  instrument  being 
held  is  of  the  type  requested.  If  so  the  instrument  being  held  shall  be  delivered  to  the 
SIU.  Otherwise,  the  Penelope  System  shall  issue  a  Free_End_Effector_Command  and 
then  continue  processing  the  Instrument_Request_Command.  This  requirement  defines  a 
special  condition  for  the  Instrument_Request_Command. 

3. 1 .2.5  Instrument_Request_Command 

Upon  receipt  of  an  Instrument_Request_Command,  if  there  are  no  special  conditions  for 
this  command,  the  Penelope  System  shall  deliver  the  requested  instrument  from  the 
instrument  cache  or  transfer  zone  to  the  SIU. 

3. 1 .2.6  Instrument  Request  Voice  Feedback 

If  so  configured,  the  Penelope  System  shall,  after  identifying  an  instrument  request,  if 
there  is  an  instrument  of  the  requested  type  that  is  not  in  use,  announce  the  instrument 
type  as  feedback  for  the  SIU. 
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3. 1 .2.7  Configurable  Instrument  Delivery  Mode 

The  Penelope  System  shall  be  configurable  so  as  to  allow  the  surgeon  to  specify  a 
preference  for  either  handoff  instrument  delivery  or  transfer  zone  instrument  delivery. 
The  preferred  delivery  method  shall  be  attempted  first. 

3. 1 .2.8  Ignored  Instrument  Requests 

If  an  SIU  requests  an  instrument  but  does  not  take  it  after  delivery,  the  Penelope  System 
shall,  after  no  less  than  <TBD:  number>  seconds,  return  the  instrument  to  the  instrument 
cache. 
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3.1.3  Instrument  Set  Requirements 

The  following  subparagraphs  list  all  functional  requirements  related  to  the  supported 
instrument  types  and  the  maintenance  of  a  list  of  instruments  used  in  a  surgical 
procedure. 

3. 1 .3. 1  Supported  Instrument  Types 

The  Penelope  System  shall  support  the  following  surgical  instrument  types. 


Large  Ricardson  Retractors 

Skin  Hooks 

Medium  Ricardson  Retractors 

Yankauer  Suction 

Small  Ricardson  Retractors 

Poole  Suction  Tip 

Baby  Ricardson  Retractors 

Frazier  Suction 

Large  Loop  Retractors 

Needle  Holder 

Small  Loop  Retractors 

Plastic  Needle  Holder 

Bull  Retractors 

Short  Sponge  Stick 

Bent  Weitlander  Retractors 

Curved  Adsons 

Plastic  Raker  Retractors 

Babcocks 

Plastic  Vein  Retractors 

Kochers 

#3  Knife  Handle 

Kellys 

#2  Knife  Handle 

Allis 

Groove 

Curved  Mosquitos 

Probe 

Criles  (Straight) 

Debakey  Forceps  5  1/2 

Curved  Criles  (Hopkins) 

Plain  Forceps 

Straight  Mayo  Scissors 

Mouth  Tooth  Forceps 

Curved  Mayo  Scissors 
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Brown  Adson  Forceps 
Plastic  Mouth  Tooth  Forceps 
Appendix  Forceps 


Metzenbaum  Scissors  Tin. 
Plastic  Straight  Scissors 
Plastic  Curved  Scissors 


3. 1 .3.2  Retain_Back_Tray_Instrument_Command 

Upon  receipt  of  a  Retain_Back_TrayJnstrument_Command,  the  Penelope  System  shall 
remove  an  instrument  of  the  specified  type  from  the  back  tray  and  add  it  to  the  instrument 
set. 

3. 1 .3.3  Retain_Added_Instrument_Command 

Upon  receipt  of  a  Retain_Added_Instrument_Command,  the  Penelope  System  shall 
ascertain  the  instrument’s  type,  move  it  from  the  receiving  zone  to  the  transfer  zone  or 
instrument  cache,  and  add  it  to  the  instrument  set. 

3. 1.3.4  Instrument  Set  Size  Warning 

Upon  receipt  of  either  a  Retain_Back_Tray_Instrument_Command  or  a 
Retain_Added_Instrument_Command,  if  the  size  of  the  instrument  set  after  the 
instrument  is  retained  is  within  <TBD:  number>  instruments  from  the  maximum  size 
specified  in  requirement  3.2. 1.3,  “Instrument  Sets,”  the  Penelope  System  shall  announce 
“<TBD:  warning>". 

3. 1.3.5  Discard  Stack 

Upon  receipt  of  either  a  Retain_Back_TrayJnstrument_Command  or  a 
Retain_Added_Instrument_Command,  if  the  size  of  the  instrument  set  before  the 
instrument  is  retained  is  equal  to  the  maximum  size  specified  in  requirement  3.2. 1.3, 
“Instrument  Sets,”  the  Penelope  System  shall  move  one  instrument  from  either  the  Mayo 
stand  or  staging  zone  to  the  discard  stack  and  then  proceed  with  the  given  command. 

3. 1 .3.6  Withdraw_Instrument_Command:  Instrument  Is  In  Use 

Upon  receipt  of  a  Withdraw_Instrument_Command,  if  an  instrument  of  the  specified  type 
is  currently  in  use,  the  Penelope  System  shall  remove  one  in  use  instance  of  that  type 
from  the  instrument  set. 

3 . 1 .3.7  Withdraw_Instrument_Command:  Instrument  Is  Not  In  Use 

Upon  receipt  of  a  Withdraw_Instrument_Command,  if  no  instrument  of  the  specified  type 
is  currently  in  use,  the  Penelope  System  shall  announce  <TBD:  warning>  . 
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3. 1.3.8  Instruinent_Count_Command 

Upon  receipt  of  an  Instrument_Count_Command,  the  Penelope  System  shall  give  a 
verbal  instrument  count  listing,  for  every  instrument  type  in  the  instrument  set,  the 
quantity  in  use,  the  quantity  in  the  ready  state,  and  the  quantity  withdrawn. 

3. 1 .3.9  Specific_Instrument_Count_Command 

Upon  receipt  of  an  Specific_Instrument_Count_Command,  the  Penelope  System  shall 
give  a  verbal  instrument  count  listing,  for  the  specified  instrument  type,  the  quantity  in 
use,  the  quantity  in  the  ready  state,  and  the  quantity  withdrawn. 

3.1.3. 10  Shared_Instrument_Count_Command 

Upon  receipt  of  a  Shared_Instrument_Count_Command,  the  Penelope  System  shall 
<TBD:  determine  exactly  how  this  procedure  between  Penelope  and  the  circulating  nurse 
will  be  carried  out.> 
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3. 1 .4  Instrument  Movement  Requirements 

The  following  subparagraphs  list  all  functional  requirements  related  to  the  movement  of 
and  handling  of  surgical  instruments  by  the  robotic  arm. 

3. 1 .4. 1  Relinquished  Instrument  Scanning 

The  Penelope  System  shall  detect,  locate,  and  identify  the  type  of  any  relinquished 
instrument  on  the  transfer  zone.  This  shall  be  accomplished  regardless  of  the  orientation 
of  the  instrument  or  whether  it  is  open  or  closed. 

3. 1 .4.2  Relinquished  Instrument  Return 

The  Penelope  System  shall  be  capable  of  moving  a  relinquished  instrument  from  any 
point  on  the  transfer  zone  to  either  the  Mayo  stand  or  the  staging  zone.  This  shall  be 
accomplished  regardless  of  the  orientation  of  the  instrument  or  whether  it  is  open  or 
closed. 

3 . 1 .4. 3  Instrument  Closing 

The  Penelope  System  shall  be  capable  of  determining  if  a  scissor-action  instrument  is 
open  and,  if  so,  closing  it  before  putting  it  on  the  Mayo  stand  or  staging  zone. 

3 . 1 .4.4  Instrument  Storage  Orientation 

The  Penelope  System  shall  be  capable  of  determining  the  orientation  of  an  instrument  on 
the  transfer  zone  and  rotating  it  so  that  it  can  be  stored  efficiently  on  the  Mayo  stand  or 
staging  zone.  This  shall  be  accomplished  regardless  of  the  orientation  of  the  instrument. 

3. 1 .4.5  Instrument  Delivery  Orientation 

When  executing  either  a  handoff  instrument  delivery  or  a  transfer  zone  instrument 
delivery,  the  Penelope  System  shall  orient  the  instrument  in  a  predefined  delivery 
orientation  that  shall  be  specified  for  each  instrument  type. 
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3.1.5  Command  Cancellation  Requirements 

The  following  subparagraphs  list  all  functional  requirements  related  to  the  cancellation  of 
command  inputs  into  the  system. 

3. 1 .5. 1  Cancel_Command:  Holding  An  Instrument 

Upon  receipt  of  a  Cancel_Command,  if  the  end  effector  is  currently  holding  an 
instrument,  the  Penelope  System  shall  issue  a  Free_End_Effector_Command  and  then 
continue  processing  the  Cancel_Command.  This  requirement  defines  a  special  condition 
for  the  Cancel_Command. 

3. 1.5.2  Cancel_Command 

Upon  receipt  of  a  Cancel_Command,  if  there  are  no  special  conditions  for  this  command, 
the  Penelope  System  shall  return  the  robotic  arm  to  its  home  position  and  disregard  any 
pending  commands  being  processed. 

3. 1 .5.3  Free_End_Effector_Command 

Upon  receipt  of  a  Free_End_Effector_Command,  if  the  end  effector  is  currently  holding 
an  instrument,  the  Penelope  System  shall  return  that  instrument  to  either  the  transfer  zone 
or  the  instrument  cache. 
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3.2  PERFORMANCE  REQUIREMENTS 

3.2.1  Safety  Requirements 

The  following  subparagraphs  list  all  performance  requirements  related  to  the  overall 
safety  of  the  Penelope  System. 

3 .2. 1 . 1  Human  Safety 

The  Penelope  System  shall  not  be  capable  of  harming  the  patient  or  any  OR  staff 
member.  This  includes  the  following. 

Harm  due  to  physical  contact  with  the  robotic  arm. 

Harm  due  to  physical  contact  with  a  surgical  instrument  being 
held  by  the  arm. 

Harm  due  to  physical  contact  with  a  surgical  instrument  that 
has  been  dropped  by  the  robotic  arm. 

Harm  due  to  any  form  of  electromagnetic  contact. 

3.2. 1 .2  Surgical  Procedure  Preclusion 

The  Penelope  System  shall  not,  under  any  operating  condition  either  nominal  or  aberrant, 
be  capable  of  precluding  the  completion  of  the  surgical  procedure.  The  complete  failure 
of  the  Penelope  System  to  operate  at  all  shall  not  preclude  the  completion  of  the  surgical 
procedure. 

3.2. 1.3  Electromechanical  Device  Safety  Standards 

The  Penelope  System  shall  be  designed  in  accordance  with  and  tested  for  compliance 
with  the  following  standards  as  they  apply  to  electromechanical  medical  devices  in 
general  and  the  Penelope  System  in  particular. 

UL  2601-1 

lEC  60601-1 

lEC  60601-1-4  Software  Collateral  Standard  <TBD:  Is  this  applicable?> 

<TBD:  Others?> 

3.2. 1 .4  Robotic  Arm  Operational  Envelop 

The  Penelope  System  shall  not,  under  any  operating  condition  either  nominal  or  aberrant, 
be  capable  of  moving  the  robotic  arm  so  that  any  portion  of  it  extends  beyond  the 


3-10 


ROBOTIC  SURGICAL  TECH,  Inc. 


PS-RSD-090803,  Draft  1 


maximum  operational  envelop.  The  perimeter  of  this  operational  envelop  shall  be 
defined  by  the  perimeter  of  the  instrument  platform  and  it  shall  extend  upward, 
perpendicular  the  surface  of  the  Mayo  stand,  to  a  height  of  <TBD:  mmber>  centimeters 
above  the  Mayo  stand. 
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3.2.2  Sterility  Requirements 

The  following  subparagraphs  list  all  performance  requirements  related  to  the  maintenance 
of  proper  sterile  technique  for  all  sterile  surfaces  and  interactions  with  the  robotic  arm. 

3.2.2. 1  Sterile  Drape  Set 

The  Penelope  System  shall  include  a  set  of  sterile  drapes  covering  all  sterile  surfaces  of 
the  system.  These  surfaces  are  defined  as  follows. 

robotic  arm 

back  tray 

receiving  zone 

Mayo  stand 

staging  zone 

transfer  zone 

upper  system  stand  assembly 

Once  draped  the  Penelope  System  shall  meet  all  sterility  requirements  listed  in  <TBD: 

OR  sterility  guidelines/requirements  document>. 

3.2.2.2  Sterile  Draping  Procedure 

The  Penelope  System  shall  define  a  sterile  draping  procedure  specifying  step-by-step 
instructions  to  be  followed  by  a  nonsterile  OR  team  member  to  apply  the  sterile  drape  set 
to  all  sterile  surfaces.  This  procedure  shall  adhere  to  all  sterility  requirements  listed  in 
<TBD:  OR  sterility  guidelines/requirements  document>. 

3.2.23  Repairing  Breaks  In  Sterile  Technique 

The  Penelope  System  shall  define  a  procedure  for  repairing  breaks  in  sterile  technique 
caused  by  unintentional  contact  between  some  part  of  the  system  and  nonsterile  personnel 
or  equipment.  Repair  procedures  shall  be  defined  for  each  the  following. 

robotic  arm 

back  tray 

receiving  zone 

Mayo  stand 
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Staging  zone 
transfer  zone 

upper  system  stand  assembly 
3. 2.2.4  Sterility  Level 

All  sterile  draping  and  sterile  materials  included  in  the  Penelope  System  shall  meet  all 
sterility  requirements  specified  in  <TBD:  some  standard.  FYI:  Aesop  out  sourced  their 
sterile  draping  to  Hydro-Med  Products,  Inc.  and  boasts  a  1996  sterility  validation  to 
ANSI/AAMIIISO  Standard  11135-1994.  They  also  claim  a  sterility  level  of  1(1^>. 
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3.2.3  Instrument  Safety  Requirements 

The  following  subparagraphs  list  all  performance  requirements  related  to  the  safe 
handling  of  surgical  instruments. 

3.2.3. 1  Instrument  Release  Height 

The  Penelope  System  shall  not,  under  any  operating  condition  either  nominal  or  aberrant, 
release  a  surgical  instrument  from  the  end  effector  from  a  height  of  more  than  <TBD. 
number>  centimeters  from  either  the  instrument  platform  or  an  SIU’s  hand. 

3.2.3.2  Instrument  Surface  Safety  Rims 

The  Penelope  System  shall  include,  on  all  sterile  surfaces  on  which  instruments  will  be 
placed,  a  safety  rim  around  the  surface’s  perimeter  raising  no  less  than  <TBD:  number> 
centimeters. 

3.2.3.3  Residual  Magnetism 

The  Penelope  System  shall  not,  through  the  handling  of  any  surgical  instrument,  induce  a 
residual  magnetism  in  that  instrument  of  sufficient  strength  to  allow  the  instrument  to  lift 
a  ferromagnetic  object  weighing  <TBD>  grams  or  more. 
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3.2.4  Anomaly  Detection  Requirements 

The  following  subparagraphs  list  all  performance  requirements  related  to  the  detection 
and  handling  of  anomalous  operating  conditions. 

3.2.4. 1  Power  Interruption 

The  Penelope  System  shall  be  capable  of  detecting  the  loss  of  electrical  power.  At  such 
time  a  Power_Down_Command  shall  be  issued  and  executed. 

3.2.4.2  Impeded  Motion  Detection 

The  Penelope  System  shall  be  capable  of  detecting  any  impediment  to  the  motion  of  the 
robotic  arm.  At  such  time  the  Penelope  System  shall  announce  “<TBD:  warning>". 
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3.2.5  Instrument  Handling  Performance  Requirements 

The  following  subparagraphs  list  all  performance  requirements  related  to  the  Penelope 
System’s  speed  and  accuracy  in  handling  surgical  instruments. 

3.2.5. 1  Instrument  Delivery  Speed:  Handoff  Delivery 

The  time  required  for  the  Penelope  System  to  deliver  an  instrument  from  the  Mayo  stand 
to  an  SIU,  when  no  instrument  is  currently  being  held  by  the  end  effector,  shall  not 
exceed  <TBD:  perhaps  3>  seconds.  This  delivery  time  shall  be  measured  from  the 
completion  of  the  instrument  request  to  the  arrival  of  the  instrument  within  the  nominal 
instrument  handoff  zone  as  defined  in  section  “<TBD>”  in  the  System  Design  Document 
(?S-SDD-<TBD>). 

3.2.5.2  Instrument  Delivery  Speed;  Transfer  Zone  Delivery 

The  time  required  for  the  Penelope  System  to  deliver  an  instrument  from  the  Mayo  stand 
to  the  transfer  zone,  when  the  end  effector  is  currently  holding  no  instrument,  shall  not 
exceed  <TBD>  seconds.  This  delivery  time  shall  be  measured  from  the  completion  of 
the  instrument  request  to  the  release  of  the  instrument. 

3.2.5. 3  Instrument  Sets 

The  Penelope  System  shall  support  instrument  sets  comprised  of  instruments  of  any  type 
defined  in  requirement  <TBD:  Paragraph  Number>,  “Supported  Instrument  Types.” 
Instrument  sets  up  to  and  including  a  maximum  of  <TBD>  instruments  shall  be 
supported. 

3.2.5.4  Instrument  Weight 

The  Penelope  System’s  robotic  arm  shall  support  instruments  up  to  and  including  a 
maximum  weight  of  <TBD>  kilograms. 

3.2.5.5  Instrument  Request  Replicability 

The  Penelope  System  shall  be  capable  of  repeating  the  process  of  identifying  an 
instrument  request,  delivering  the  instrument  to  the  SIU,  identifying  that  instrument  when 
it  is  relinquished,  and  returning  it  to  the  Mayo  stand  through  no  less  than  <TBD: 
number>  iterations  without  any  cumulative  degradation  of  accuracy. 

3.2.5.6  False-Positive  Relinquished  Instrument  Identification 

The  Penelope  System  shall  achieve  a  false-positive  error  rate  for  relinquished  instrument 
identification  of  no  more  the  <TBD>%.  A  false-positive  relinquished  instrument 
identification  error  is  defined  as  the  perception  by  the  system  that  a  relinquished 
instrument  is  on  the  transfer  zone  when  that  instrument  is  not  in  fact  present.  This  also 
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includes  the  case  where  a 
system  identifies. 


valid  instrument  is  on  the  transfer  zone,  but  not  the  one  the 


3.2.5.7  True-Negative  Relinquished  Instrument  Identification 


The  Penelope  System  shall  achieve  a  true-negative  error  rate  for  relinquished  instrument 
identification  of  no  more  the  <TBD>%.  A  true-negative  relinquished  instrument 
identification  error  is  defined  as  the  failure  of  the  system  to  correctly  identify  a 
relinquished  instrument  on  the  transfer  zone  that  is  known  to  be  in  use. 
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3.2.6  Verbal  Command  Recognition  Performance  Requirements 

The  following  subparagraphs  list  all  performance  requirements  related  to  the  accuracy  of 
the  Penelope  System’s  speech  recognition  system. 

3.2.6. 1  False-Positive  Verbal  Command  Recognition 

The  Penelope  System  shall  achieve  a  false-positive  error  rate  for  verbal  command 
recognition  of  no  more  the  <TBD>%.  A  false-positive  verbal  command  recognition 
error  is  defined  as  the  recognition  by  the  system  of  a  verbal  command  when  that 
command  was  not  given.  This  also  includes  the  case  where  a  valid  verbal  command  was 
given  but  was  incorrectly  identified  by  the  system. 

3.2.6.2  True-Negative  Verbal  Command  Recognition 

The  Penelope  System  shall  achieve  a  true-negative  rate  for  verbal  command  recognition 
of  no  more  the  <TBD>%.  A  true-negative  verbal  command  recognition  is  defined  as  the 
failure  of  the  system  to  recognize  a  verbal  command. 
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3.2.7  Environmental  Requirements 

The  following  subparagraphs  list  all  performance  requirements  related  to  the  operational 
conditions  under  which  the  Penelope  System  must  function. 

3.2.7. 1  Electrical  Power 

The  Penelope  System  shall  operate  on  AC  power  at  any  voltage  from  <TBD:  maybe 
108>  volts  to  <TBD:  maybe  132>  volts. 

3. 2.7.2  Noise  Level 

The  Penelope  System  shall  be  capable  of  meeting  all  verbal  command  recognition 
requirements  in  an  environment  with  an  ambient  noise  level  up  to  <TBD.  number> 
decibels. 

3.2.7.3  Lighting  Level 

The  Penelope  System  shall  be  capable  of  meeting  all  instrument  identification  and 
handling  requirements  in  an  environment  with  an  ambient  light  level  as  low  as  <TBD. 
number>  lumens  per  square  meter. 

3. 2.7.4  Splash  Resistance 

The  Penelope  System  shall  be  capable  of  withstanding  splash  exposure  to  as  much  as 
<TBD:  number>  liters  of  blood,  bodily  fluids,  or  disinfectant  solution  without  sustaining 
any  performance  reducing  damage. 
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3.2.8  Structural  Requirements 

The  following  subparagraphs  list  all  performance  requirements  related  to  the  structural 
integrity  of  the  Penelope  System. 

3.2.8. 1  System  Stand  Strength  and  Stability 

The  system  stand  of  the  Penelope  System,  with  all  functional  components  attached,  shall 
capable  of  supporting,  without  tipping  over  or  sustaining  any  structural  damage,  a  weight 
of  no  less  than  <TBD;  number>  kilograms  placed  at  any  point  on  the  instrument 
platform. 

3.2.8.2  Total  Weight 

The  total  weight  of  the  Penelope  System,  with  all  functional  components  attached,  shall 
not  exceed  <TBD:  number>  kilograms. 

3.2.8.3  Impact  Resistance 

The  Penelope  System  shall  be  capable  of  withstanding,  without  sustaining  any 
performance  reducing  damage,  an  impact  from  a  mass  of  up  to  <TBD:  number> 
kilograms  traveling  at  a  velocity  of  up  to  <TBD:  nuTnbev>  meters  per  second.  This 
impact  shall  be  withstood  when  applied  to  any  exposed  surface  of  the  Penelope  System. 
<TBD:  Need  a  more  precise  way  of  expressing  this,  possibly  in  terms  of  momentum.> 
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3.3  INTERFACE  REQUIREMENTS 

3.3. 1  System  Control  Input  Requirements 

The  following  subparagraphs  list  all  interface  requirements  related  to  the  input  of 
commands  into  the  Penelope  System. 

3 . 3 . 1 . 1  V  oice  Commands 

The  Penelope  System  shall  be  capable  of  detecting  and  the  activating  of  the  following 
commands  through  voice  input. 

Instrument_Request_Command 

Cancel_Command 

Instrument_Count_Command 

Specific_Instrument_Count_Command 

Shared_Instrument_Count_Command 

Retain_Added_Instrument_Command 

Withdraw_Instrument_Command 

3. 3 . 1 .2  Master  Power  Switch 

The  Penelope  System  shall  include  a  power  switch,  which  either  engages  or  interrupts 
power  to  all  system  components. 

3.3. 1 .3  Halt/Resume  Button 

The  Penelope  System  shall  include  on  the  control  panel  a  toggle  button  or  switch  via 
which  an  OR  team  member  can  alternatively  issue  either  a  Halt_Command  or  a 
Resume_Command. 

3.3.1 .4  Power  Down  Button 

The  Penelope  System  shall  include  on  the  control  panel  a  button  via  which  an  OR  team 
member  can  issue  a  Power_Down_Command. 
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3.3.2  System  Mode  Feedback  Requirements 

The  following  subparagraphs  list  all  interface  requirements  related  to  the  output  of 
system  mode  information  from  the  Penelope  System. 

3.3.2. 1  Power  Indicator 

The  Penelope  System  shall  include  an  visible  indicator  signifying  that  the  system  is 
powered. 

3.3.2.2  Verbal  Command  Input  Indicator 

The  Penelope  System  shall  include  an  visible  indicator  signifying  that  the  system  is 
listening  for  voice  commands. 
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3.3.3  Mobility  And  Positioning  Requirements 

The  following  subparagraphs  list  all  interface  requirements  related  to  moving  the 
Penelope  System  around  and  positioning  it  prior  to  a  surgical  procedure. 

3.3.3. 1  In-Hospital  System  Stand  Transportability 

It  shall  be  possible  for  one  person  to  accomplish  in-hospital  transportation  all  functional 
components  of  the  Penelope  System  across  a  level,  flat  surface. 

3.3.3.2  Robotic  Arm  In-Transit  Tie-down 

The  Penelope  System  shall  include  a  mechanism  for  securing  the  robotic  arm  during 
transportation. 

3.3.3.3  Cabling  In-Transit  Tie-down 

The  Penelope  System  shall  include  a  mechanism  for  securing  all  required  cabling  during 
transportation. 

3.3.3.4  System  Stand  In-Transit  Envelop 

During  in-hospital  transportation,  all  functional  components  of  the  Penelope  System  shall 
be  no  more  than  <TBD:  number>  centimeters  wide  and  <TBD:  nMm*er>centimeters  tall. 

3.3.3.5  System  Stand  Positioning  Lock-down 

The  Penelope  System  shall  include  a  mechanism  for  locking  the  system  stand  down  so 
that  it  cannot  move.  The  Penelope  System  shall  also  include  a  mechanism  for  unlocking 
the  system  stand. 

3. 3. 3.6  System  Stand  Positioning  Over  OR  Table 

The  Penelope  System  shall  include  a  mechanism  for  positioning  the  system  stand  at  any 
point  over  the  operating  table. 

3.3.3.7  System  Stand  Height  Adjustment 

The  Penelope  System  shall  include  a  mechanism  for  adjusting  the  height  of  the  system 
stand  from  <TBD:  rtMmfeer>centimeters  to  <TBD:  number>c&aAmsX&rs  above  the  floor. 
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3.3.4  Field  Maintenance  Requirements 

The  following  subparagraphs  list  all  interface  requirements  related  to  maintenance  of  the 
Penelope  System  after  it  has  been  deployed. 

3.3.4. 1  Field  Repairable  Components 

The  following  system  components  shall  be  repairable  by  qualified  service  personnel  at 
the  installation  site. 

robotic  arm 

instrument  deck 

camera  mount  assembly 

<TBD:  Identify  these  components.  This  is  essentially  the  list  of  replacement  parts 
a  technician  would  carry  around.> 

3.3.4.2  System  Diagnostics 

The  Penelope  System  shall  include  a  set  of  diagnostic  tests  that  can  be  run  by  a  qualified 
service  representative.  These  tests  shall  confirm  that  all  essential  components  of  the 
system  are  functioning  in  accordance  with  their  specifications.  <TBD:  list  which 
components  must  be  tested.> 

3.3.4.3  System  Software  Upgrades 

The  Penelope  System  shall  include  a  mechanism  allowing  qualified  service  personnel  to 
upgrade  portions  of  the  system  software  as  required.  Any  such  update  shall  not 
invalidate  or  corrupt  any  site-specific  data  stored  for  the  installation.  <TBD:  list  types  of 
software  upgrades.> 
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